Methods and apparatuses for providing procedure guidance

ABSTRACT

Apparatuses and methods of operating the same are described. An apparatus including a display, an input device, and a processing device coupled to the display and the input device. The processing device may send an output to the display. The output may include a graphical object associated with a first step of a user-implemented procedure. The processing device may receive an input from the input device. The input may indicate a progress on an execution of the first step by an operator. The processing device may determine whether the input indicates that the operator has completed the first step. The processing device may determine whether the first step is a final step in the user-implemented procedure. The processing device may identify a second step in the user-implemented procedure when the input indicates that the operator has completed the first step and the first step is not a final step.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/001,652, titled “METHODS AND APPARATUSES FOR PROVIDING PROCEDUREGUIDANCE,” filed Aug. 24, 2020, now U.S. Pat. No. 11,592,897, which is acontinuation of U.S. patent application Ser. No. 15/839,003, titled“METHODS AND APPARATUSES FOR PROVIDING PROCEDURE GUIDANCE,” filed Dec.12, 2017, now U.S. Pat. No. 10,782,775, which claims the benefit of U.S.Provisional Application No. 62/446,012, filed Jan. 13, 2017.

BACKGROUND

Guidance for and/or confirmation of a procedure being carried out by (orpartly by) an operator may be facilitated in a number of ways. Forexample, a paper document such as a maintenance checklist may beprepared beforehand, with instructions for each event that is to beexecuted, a box to be checked or initialed by the operator to confirmevents have been executed, blanks to record data or comments associatedwith events in the procedure, etc. However, such static instructions anddocumentation may exhibit limitations in at least some instances.

For example, linear paper checklists may not closely correspond withprocedures that may be non-linear. As a more concrete example, inperforming maintenance on complex systems such as aircraft engines, theappropriate procedure may branch, loop back, or be otherwise non-lineardepending on the condition of the engine itself; a paper checklist maynot accommodate adding a sequence to lubricate some engine component ifit is determined to be necessary upon inspection, skipping replacementof a presumed worn part if the part is in fact not worn, etc.

In addition, simple paper checklists or similar may be limited in termsof the guidance that may be offered, and/or the feedback that may beaccepted. To continue the example above, a paper checklist may lackspace for detailed drawings showing the individual engine components tobe serviced or for a specific maintenance history for the particularengine being serviced, may not support evidence or confirmation that anew component has been installed properly beyond a check-off, sign-off,etc.

Furthermore, static guidance and feedback may not adapt well to changingconditions. Modest changes to an engine or to “best practices” in enginemaintenance may render entire checklists (or at least significantportions thereof) obsolete. Appropriate procedures may even varydepending on the age or history of a particular engine, e.g. a10-year-old engine may be prone to different maintenance issues than anominally-identical newer engine, an engine that has shown an unusualhistory of wear to certain components may benefit from greater attentionto those or related components for that specific engine, etc.

SUMMARY

An apparatus including a display, an input device, and a processingdevice coupled to the display and the input device. The processingdevice may send an output to the display. The output may include agraphical object associated with a first step of a user-implementedprocedure. The processing device may receive an input from the inputdevice. The input may indicate a progress on an execution of the firststep by an operator. The processing device may determine whether theinput indicates that the operator has completed the first step. Theprocessing device may determine whether the first step is a final stepin the user-implemented procedure. The processing device may identify asecond step in the user-implemented procedure when the input indicatesthat the operator has completed the first step and the first step is nota final step.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart for confirming the execution of a procedurewith regard to a script operator, according to an embodiment.

FIG. 2A shows a flowchart for the arranging of a script for guiding aprocedure and confirming execution, according to an embodiment.

FIG. 2B shows another flowchart for the arranging of a script forguiding a procedure and confirming execution, according to anembodiment.

FIG. 3 shows a script is shown in flowchart form for the arranging of ascript for guiding a procedure and confirming execution, according to anembodiment.

FIG. 4A shows a flowchart for the arranging of a script for guiding aprocedure and confirming execution with regard to a script author,according to an embodiment.

FIG. 4B shows a continuation of the flowchart from FIG. 4A for thearranging of a script for guiding a procedure and confirming executionwith regard to a script author, according to an embodiment.

FIG. 5A shows an interface for authoring a script, according to anembodiment.

FIG. 5B shows an interface for executing a script, according to anembodiment.

FIG. 6 shows a flowchart for providing guidance during a procedure andfor confirming the execution of that procedure with regard to ascripting architect, according to an embodiment.

FIG. 7 shows an apparatus for providing guidance for or during aprocedure and for confirming the execution of that procedure with regardto a scripting operator, according to an embodiment.

FIG. 8 shows an apparatus for providing guidance for or during aprocedure and for confirming the execution of that procedure with regardto a scripting author, according to an embodiment.

FIG. 9 shows an apparatus for providing guidance for or during aprocedure and for confirming the execution of that procedure, accordingto an embodiment.

FIG. 10 shows a block diagram is shown of an apparatus that may performvarious operations, and store various information generated and/or usedby such operations, according to an embodiment.

DETAILED DESCRIPTION

A variety of systems, apparatus, methods, and paradigms for providingguidance for executing a procedure, and for confirming the execution ofthe procedure, are contemplated herein. In a broad and non-limitingoverview, a procedure to be performed is guided (and potentiallyrepresented) by a script. The script is configured so that executing thescript substantially corresponds to executing the procedure.

It is noted that certain parts of the script may exist as data and/orexecutable instructions, for example as may be carried out by aprocessor in an electronic device such as a smartphone, head mounteddisplay, etc. However, scripts typically include outputs to an operatorperforming the procedure (e.g. by following the script), inputs fromthat operator, outputs and inputs to or from other actors, commandsand/or data to or from sensors, communications devices, and/or otherhardware, etc. Moreover, the actions taken by an operator following thescript typically are “real world” physical actions, such as servicing amachine, disassembling or reassembling some device, etc. Thus while thescript may include processor-implemented features, the script and use ofthe script overall are not exclusively processor-implemented.

Again with regard to a broad but non-limiting overview, a scriptincludes steps that correspond with various events for the procedure,such as instructions for tasks to be performed as part of the procedure,diagrams or other information supporting proper performance of theprocedure, data to be taken in performing the procedure, evidence to berecorded showing that the procedure has been performed properly, etc.Links within the script connect various steps so that one step followsanother in an order suited to the procedure in question.

Such an arrangement of steps and links in a script may be at leastsomewhat analogous to a flowchart. However, where a fixed paperflowchart may be essentially “dead”, a script may be interactive withthe operator (and/or potentially other persons or systems) such as byproviding outputs and accepting inputs, and also may be adaptive tolocal conditions such as by branching, looping, contacting a remoteexpert, etc. depending on how a particular performance of the procedureunfolds. Thus in some sense, a script may be considered as an “activeflow chart”.

With regard to particulars and examples of providing guidance forexecuting a procedure and confirming the execution of the procedure, itmay be useful to provide brief explanations of a number of terms.

An “operator” refers to a person performing some task. For example, anaircraft mechanic may be an operator for a task associated with therepair or maintenance of an aircraft. Operators may vary greatly and arenot limited. In addition, it is not required for an operator to performall actions associated with a given task; for example, if an aircraftmechanic is repairing a landing gear photographs may be takenautomatically as the mechanic works, without the mechanic necessarilytriggering those photographs. Two or more operators may be associatedwith a given task, or with certain actions that are part of that task.In principle, it may be possible for certain tasks to be performedentirely without an operator, though for purposes of example herein itis typically assumed that an operator is present and participating.

A “procedure” refers to the performance of some task. For example,inspecting or replacing the tires on an aircraft landing gear may be aprocedure. Typically though not necessarily a procedure is relativelycomplex, with many individual actions, but no particular limits areplaced on the degree of complexity. For many (though not necessarilyall) procedures addressed herein, at least some part of the procedure iscarried out by an operator (as distinct from being fully automated,taking place entirely within a processor, etc.).

An “event” refers to some particular action or other portion of aprocedure. For example, “loosen the bolts to remove the cover”, “examinethe tire tread”, or “check the serial number for the landing gearassembly” may all be events in a procedure. As with a procedure overall,events thereof are not limited with regard to a degree of complexity.However, typically (though not necessarily) an event may be relativelysimple, and/or brief, e.g. one action in assembling a device, one or agroup of a few related inspections, a well-defined test sequence, etc.

A “script” refers to a set of operator instructions, processorinstructions, information, feedback, options, etc. that correspond witha procedure, such that executing the script at least substantiallycorresponds to executing the procedure. The degree of correspondence isnot limited, nor is perfect or total correspondence with the procedurerequired. A script may exclude and/or summarize portions of a procedure,for example a script that includes a step for tightening bolts may notindividually address each bolt, may not provide specific instructions onhow to tighten bolts, may not specify that a wrench is to be used, etc.(though such specificity is not prohibited). It may be suitable to omitdetails and actions so long as the procedure reasonably may beunderstood, e.g. so that an operator would understand and be able tocarry out a procedure by following a script for that procedure. Thepresumed capabilities of operators may vary; a script for a trainedaircraft mechanic may presume a high level of experience with aircraftrepair generally, and may not include the same level of detail as ascript for an emergency field repair by an untrained individual.Likewise, a script for repairing plumbing directed to a new homeownermay presume little or no knowledge of plumbing and thus be extremelydetailed, while a script for the same repair procedure directed to aprofessional plumber may presume much greater knowledge of plumbing andbe much less detailed. Conversely, however, a script that presumesgreater skill may, in fact, be more complex, for instance includingadditional options; to continue the example of the plumbing repair, ascript for a professional plumber may prompt the plumber to considerwhich of several varieties of repair would be most suitable (mosteconomical, quickest, sturdiest, etc.), while a script for a newhomeowner may simply provide a “one size fits all” solution on theassumption that the operator, in that case, may not have the experienceto make a judgment as to what repair is suitable. The degree to which agiven script should correspond to a given procedure thus is to at leastsome extent variable, as may be understood by a person of skill in theart.

A “step” refers to a portion of a script addressing some action,typically though not necessarily corresponding with an event in aprocedure. For example, if an event in a procedure is tightening thebolts on some piece of hardware, a step corresponding with that eventmay include instructions to tighten the bolts, an augmented realityfeature pointing out the bolts, an indication of which size wrench maybe appropriate, etc. However, it is emphasized that steps in a script(and thus a script overall) are not limited only to providinginformation. A step may include additional features, including receiptof data (e.g. the torque values used to tighten the bolts), actions nottaken by the operator and/or not visible to the operator (e.g.automatically taking a photograph of the assembly that may show whetherthe bolts are properly installed), communication with other parties(e.g. a referencing a database to call up a schematic for a particulardevice, contacting an expert in a machine, a task, etc., a supervisor orinspector to “sign off” on some action, etc.), storing or retrievingdata, data processing (e.g. analyzing an image of a machine to determineif it is the machine that is to be worked on), etc. A step also mayrepresent a decision to be made; for example, if step A could befollowed by either step B1 or step B2, determining whether to proceed tostep B1 or step B2 may be addressed as a step in itself In addition,although certain steps may include interaction with and/or actions takenby an operator, not all steps necessarily will; certain steps may, forexample, be executed entirely automatically, e.g. by a processor, by asensor, etc. To continue the example above with regard to the decisionstep for determining to follow step B1 or B2, although in some instancesthe operator may make that decision, in other instances the decision maybe made automatically (and thus from their standpoint the operator“doesn't do anything” for that decision step).

A “link” refers to a portion of a script that indicates sequencing amongsteps, e.g. “proceed from step 6 to step 7”. Typically though notnecessarily links in a script corresponding to a practical sequencing ofevents in a procedure. While links themselves may not be or equate toactions, it may be understood that the arrangement of links among stepsin some sense defines steps into a script, rather than a mere collectionof unrelated instructions. That is, A, B, and C may be mere steps;linking those steps into an arrangement that begins with A, thenproceeds to B, then moves on to C may assemble those steps into ascript. In addition, it is noted that while in certain examples linksmay be arranged in linear fashion, such that steps form a script thatmight be expressed as A→B→C→D→E . . . , links also may be arranged inmore complex and/or non-linear fashions, for example A→(B1 orB2)→C→D→(B1 or B2) . . . and so forth. The arrangement of links (andthus the overall organization of a script) may vary greatly, and atleast in principle may be arbitrarily large and/or complex.

A “cue” is a particular feature of a step that is specified in a script.For example, an output cue may specify output that is to be deliveredfor a step, when, how, in what form, etc. An input cue may specify inputthat is to be received for a step, through what avenue (e.g. aphotograph from a camera, text or voice input, etc.), in what form, andso forth. Typically though not necessarily, each step in a script willhave at least one cue associated therewith. Steps may have multipleinputs associated therewith, including cues of the same general type(e.g. several output cues) and cues of different types (e.g. an outputcue and an input cue). In addition, a single cue may be associated withmultiple steps. For example, an output cue for delivering a“boilerplate” warning to an operator that components are hot or that agrounding strap must be worn by may be associated with several or evenall steps in a script, thus being executed again for each such step.

It is noted that an output cue is not necessarily identical to theoutput itself. For example, an output may be a schematic for a devicethat is being repaired, while the output cue may specify that theschematic is to be displayed graphically on a display, that theschematic is to be positioned in a particular portion of the display,that the schematic is to be shown only in response to a verbal commandfrom the user (e.g. the user says “schematic”), etc. Likewise, while adata retrieval cue to access that schematic from stored memory is insome sense addressing output (namely the schematic), again the cuereferring to the output and the output itself are not identical.

With regard to “output cue” and “input cue”, the directions of “in” and“out” refer to the script. Thus an output cue addresses something (e.g.information) moving from the script (e.g. instructions to the operator),while an input cue addresses something moving to the script (e.g. datafrom a sensor). Although useful for illustration, it should beunderstood that such an arrangement may not always be literally true;for example an image that is to be delivered as an output for an outputcue may not literally exist within the script (instead being an imagefile that is called from a hard drive, etc.), and thus may not literallycome from or originate in the script (although literally including theimage file as part of the script itself also is not prohibited).

In addition, it should be understood that certain of the definitionsused herein may be at least somewhat flexible. For example, differentscripts may consider removing each of several bolts to be individualevents (potentially with different steps for each bolt), but mayalternately consider removing all bolts a single event, or againalternately may consider exposing a machine for maintenance by removingnumerous bolts, covers, etc. a single event, etc. While some degree ofconsistency within a procedure and among related or similar proceduresmay be desirable, no particular precise or universal limits are imposedon what “counts” as a step, an event, a procedure, etc. so long thenecessary functionality is enabled.

The embodiments are directed to a variety of systems, apparatus,methods, and paradigms for providing guidance for executing a procedure,and for confirming the execution of the procedure.

In one embodiment, a method is provided that includes instantiating ascript for a user-implemented procedure onto a processor of a portableelectronic device. The script includes script steps with each stepcorresponding with an event of the procedure, and script linkscorresponding with sequencing the events and with each step except thefinal step being engaged with a subsequent step by at least one links.Each step includes an output cue associated with the script step andspecifying a graphical output from the processor to the operator for thestep, the graphical output including guidance to the operator forexecuting the event of the procedure associated with the step. Each stepalso includes an input cue associated with the script step andspecifying a graphical input from the operator to the processor for thestep, the input providing feedback regarding the operator executing theevent of the procedure associated with the step. The method includesexecuting an initial step: executing the output cue including deliveringthe graphical output and executing the input cue including accepting theinput. If the input cue and the output cue are satisfied, and the stepis not the final step, the link is followed to the subsequent step.Executing the script substantially corresponds to executing theprocedure.

In another embodiment, a method is provided that includes instantiatinga script for a procedure onto a processor, the procedure being at leastpartially operator-implemented. The script includes steps correspondingto events of the procedure and links corresponding with sequencing theevents, with each step except a final step being engaged with asubsequent step by at least one link. The script includes cues, each cuebeing associated with at least one step and each step having at leastone cue associated therewith. The cues include output cues specifyingoutput from the processor, at least one output cue specifying the outputbeing to the operator, the output to the operator including guidance tothe operator for executing the procedure. The cues also include inputcues specifying input to the processor, at least one input cuesspecifying the input being from the operator, the input from theoperator providing feedback regarding the operator executing theprocedure. The method includes executing an initial step: executing theoutput cue if the step has an output cue associated therewith, andexecuting the input cue if the step has an input cue associatedtherewith. If the input cue and the output cue are satisfied, and thestep is not the final step, the link is followed to the subsequent step.Executing the script at least substantially corresponds to executing theprocedure.

The output specified by the output cue may include text, a schematic, agraphic, an image, an animation, a video, audio, augmented realitycontent, virtual reality content, stereo content, an executableinstruction, and/or an external communication.

The output cue may execute automatically. The output cue may executeresponsive to a timer, a user command, and/or an environmental factor.

An output cue may specify the output being to a recipient other than theoperator.

Delivering the output from the processor may satisfy the output cue.Satisfying the output cue may include confirming delivery of the output.

The input specified by the input cue may include text, an image, avideo, audio, augmented reality content, virtual reality content, stereocontent, an executable instruction, an external communication, aselection input, a machine-readable code, and/or sensor data.

The input cue may execute automatically.

The output cue may execute responsive a timer, a user command, and/or anenvironmental factor.

Receiving the input to the processor may satisfy the input cue.Satisfying the input cue may include confirming a receipt of the input,confirming a presence of content in the input, confirming a type of thecontent in the input, confirming a format of the content in the input,confirming the content in the input satisfies an input standard, and/orconfirming a presence of a feature in the content in the input.

The cues may also include an identity cue specifying establishing anidentity of the operator, the processor and/or the script. The cues mayalso include an operator communication cue specifying establishing acommunication between the operator and an entity other than the operatorand the processor. The cues may also include a processor communicationcue specifying establishing a communication between the processor and anentity other than the operator and the processor. The cues may alsoinclude a processing cue specifying data analysis of the input and/orthe output within the processor. The cues may also include a storage cuespecifying storage of data by the processor. The cues may also include aretrieval cue specifying access of stored data by the processor.

The steps and links may be configured such that the script isnon-linear. At least one of the steps may be engaged with multiple linksto multiple subsequent steps. At least one step may include multiplecues.

In another embodiment, an apparatus is provided that includes a bodyadapted to be worn on a head of an operator, a processor engaged withthe body, a stereo graphical display in communication with the processorand disposed on the body such that when the body is worn by the operatorthe graphical display is facing and proximate eyes of the operator, adepth camera in communication with the processor and disposed on thebody such that when the body is worn by the operator the depth camerafaces at least substantially forward with respect to the head of theoperator, and a color camera in communication with the processor anddisposed on the body such that when the body is worn by the operator thecolor camera faces at least substantially forward with respect to thehead of the operator. The apparatus includes a script for auser-implemented procedure instantiated on the processor. The scriptincludes script steps with each step corresponding with an event of theprocedure, and script links corresponding with sequencing the eventswith each step except a final step being engaged with a subsequent stepby at least one link. Each step includes an output cue associated withthe script step and specifying a graphical output from the processor tothe operator for the step, the graphical output including guidance tothe operator for executing the event of the procedure associated withthe step. Each step includes an input cue associated with the scriptstep and specifying a graphical input from the operator to the processorfor the step, the input providing feedback regarding the operatorexecuting the event of the procedure associated with the step. Theprocessor is adapted to execute the script beginning with an initialstep. Executing the steps includes executing the output cue includingdelivering the graphical output to the operator via the stereographicaldisplay, and executing the input cue including accepting the input fromthe operator via at least one of the depth camera and the color camera.If the input cue and the output cue are satisfied, and the step is notthe final step, the link is followed to the subsequent step. Executingthe script substantially corresponds to executing the procedure.

In another embodiment, an apparatus is provided that includes aprocessor, an outputter in communication with the processor, an inputterin communication with the processor, and a script for a procedureinstantiated on the processor, the procedure being at least partiallyoperator-implemented. The script includes steps corresponding withevents of the procedure, links corresponding with sequencing the eventswith each of the steps except a final step being engaged with asubsequent step by at least one of the links, and cues with each cuebeing associated with at least one step and each step having at leastone cue associated therewith. The cues include an output cue specifyingan output from the processor with at least one of the output cuesspecifying the output being to the operator and the output to theoperator including guidance to the operator for executing the procedure,and an input cue specifying an input to the processor with at least oneof the input cues specifying the input being from the operator and theinput from the operator providing feedback regarding the operatorexecuting the procedure. The processor is adapted to execute the scriptbeginning with an initial step. Executing the steps includes executingthe output cue if the step has an output cue associated therewith,executing the input cue if the step has an input cue associatedtherewith, and if the input cue and the output cue are satisfied and thestep is not the final step, following the link to the subsequent step.Executing the script at least substantially corresponds to executing theprocedure.

The outputter may include a graphical display, a CRT display, an LEDdisplay, an OLED display, a plasma display, a stereo display, asee-through display, an audio speaker, stereo audio speakers, and/or awireless data link. The inputter may include a still imager, a videoimager, a visible light imager, a near-infrared imager, a thermalinfrared imager, a depth imager, a microphone, and/or a wireless datalink. The inputter may be adapted to accept gesture input.

The apparatus may include a body adapted to be worn on the head of theoperator, with the processor, the outputter, and the inputter engagedwith the body.

In another embodiment, an apparatus is provided that includes means forinstantiating a script for a procedure onto a processor, the procedurebeing at least partially operator-implemented. The script includes stepscorresponding with events of the procedure, links corresponding withsequencing the events with each step except a final step being engagedwith a subsequent step by at least one link, and cues with each cuebeing associated with at least one of the steps and each step having atleast one cue associated therewith. The cues include output cuesspecifying output from the processor with at least one output cuespecifying the output being to the operator and the output to theoperator including guidance to the operator for executing the procedure,and input cues specifying input to the processor with at least one inputcue specifying the input being from the operator and the input from theoperator providing feedback regarding the operator executing theprocedure. The method also includes means for executing an initial stepof the steps. Executing the steps includes executing the output cue ifthe step has an output cue associated therewith, executing the input cueif the step has an input cue associated therewith, and if the input cueand the output cue are satisfied and the step is not the final stepfollowing the link to the subsequent step. Executing the script at leastsubstantially corresponds to executing the procedure.

In another embodiment, a method is provided that includes authoring ascript for a procedure onto a processor of a portable electronic device.The script includes script steps corresponding to events of theprocedure, and script links corresponding to sequencing the events witheach step except a final step being engaged with a subsequent step by atleast one link. Each step includes an output cue associated with thescript step and specifying a graphical output from the processor to theoperator for the step, the graphical output including guidance to theoperator for executing the event of the procedure associated with thestep. Each step also includes an input cue associated with the scriptstep and specifying a graphical input from the operator to the processorfor the step, the input providing feedback regarding the operatorexecuting the event of the procedure associated with the step.

Authoring the script includes an author disposing a graphical stepavatar class on the processor, and disposing a step class associatedwith the step avatar class on the processor such that instantiating astep avatar from the step avatar class instantiates the step from thestep class on the processor therewith. Authoring includes the authordisposing a graphical link avatar class on the processor, and disposinga link class associated with the link avatar class on the processor suchthat instantiating a link avatar from the link avatar class instantiatesthe link from the link class on the processor therewith. Authoring alsoincludes the author disposing a graphical output cue avatar class on theprocessor and disposing an output cue class associated with the outputcue avatar class on the processor such that instantiating an output cueavatar from the output cue avatar class instantiates the output cue fromthe output cue class on the processor therewith. Authoring includes theauthor disposing a graphical input cue avatar class on the processor,and disposing an input cue class associated with the input cue avatarclass on the processor such that instantiating an input cue avatar fromthe input cue avatar class instantiates the input cue from the input cueclass on the processor therewith. Authoring further includes the authorinstantiating step avatars in a graphical interface for the processor soas to instantiate steps on the processor, instantiating link avatars inthe graphical interface so as to instantiate links on the processor, andinstantiating cue avatars in the graphical interface for the processorso as to instantiate cues on the processor. Executing the scriptsubstantially corresponds to executing the procedure.

In another embodiment, a method is provided that includes authoring ascript for a procedure onto a processor of a portable electronic device,the procedure being at least partially operator-implemented. The scriptincludes steps corresponding with events of the procedure, linkscorresponding with sequencing the events with each step except a finalstep being engaged with a subsequent step by at least one link, and cueswith each the cue being associated with at least one step and each stephaving at least one cue associated therewith. The cues include outputcues specifying output from the processor, at least one output cuespecifying the output being to the operator, the output to the operatorincluding guidance to the operator for executing the procedure. The cuesalso include input cues specifying input to the processor, at least oneinput cue specifying the input being from the operator, the input fromthe operator providing feedback regarding the operator executing theprocedure.

Authoring the script includes an author disposing a graphical stepavatar class on the processor, and disposing a step class associatedwith the step avatar class on the processor such that instantiating astep avatar from the step avatar class instantiates the step from thestep class on the processor therewith. Authoring the script includes theauthor disposing a graphical link avatar class on the processor, anddisposing a link class associated with the link avatar class on theprocessor such that instantiating a link avatar from the link avatarclass instantiates the link from the link class on the processortherewith. Authoring the script also includes the author disposing agraphical output cue avatar class on the processor, and disposing anoutput cue class associated with the output cue avatar class on theprocessor such that instantiating an output cue avatar from the outputcue avatar class instantiates the output cue from the output cue classon the processor therewith. Authoring the script includes the authordisposing a graphical input cue avatar class on the processor anddisposing an input cue class associated with the input cue avatar classon the processor such that instantiating an input cue avatar from theinput cue avatar class instantiates the input cue from the input cueclass on the processor therewith. Authoring the script further includesthe author instantiating step avatars in a graphical interface for theprocessor so as to instantiate steps on the processor, instantiatinglink avatars in the graphical interface so as to instantiated links onthe processor, and instantiating cue avatars in the graphical interfacefor the processor so as to instantiate cues on the processor. Executingthe script at least substantially corresponds to executing theprocedure.

The author may specify the output cue as including text, a schematic, agraphic, an image, an animation, a video, audio, augmented realitycontent, virtual reality content, stereo content, an executableinstruction, and/or an external communication.

The author may specify the output cue as executing automatically. Theauthor may specify the output cue as executing responsive to a timer, auser command, and/or an environmental factor.

The author may specify the output for at least one output cue as beingto a recipient other than the operator.

The author may specify delivering the output from the processor assatisfying the output cue. The author may specify that satisfying theoutput cue includes confirming delivery of the output.

The author may specify that the input for the input cue includes text,an image, a video, audio, augmented reality content, virtual realitycontent, stereo content, an executable instruction, an externalcommunication, a selection input, a machine-readable code, and/or sensordata.

The author may specify the input cue as executing automatically. Theauthor may specify the output cue as executing responsive to a timer, auser command, and/or an environmental factor.

The author may specify that receiving the input to the processorsatisfies the input cue. The author may specify that satisfying theinput cue includes confirming a receipt of the input, confirming apresence of content in the input, confirming a type of the content inthe input, confirming a format of the content in the input, confirmingthe content in the input satisfies an input standard, and/or confirminga presence of a feature in the content in the input.

The cues may further include an identity cue specifying establishing anidentity of the operator, the processor, and/or the script, an operatorcommunication cue specifying establishing a communication between theoperator and an entity other than the operator and the processor, aprocessor communication cue specifying establishing a communicationbetween the processor and an entity other than the operator and theprocessor, a processing cue specifying data analysis of at least one ofthe input and the output within the processor, a storage cue specifyingstorage of data by the processor, and/or a retrieval cue specifyingaccess of stored data by the processor.

The method may include the author disposing a graphical identity cueavatar class on the processor, and disposing an identity cue classassociated with the identity cue avatar class on the processor such thatinstantiating an identity cue avatar from the identity cue avatar classinstantiates the identity cue from the identity cue class on theprocessor therewith. The method may include the author disposing agraphical operator communication cue avatar class on the processor, anddisposing an operator communication cue class associated with theoperator communication cue avatar class on the processor such thatinstantiating an operator communication cue avatar from the operatorcommunication cue avatar class instantiates the operator communicationcue from the operator communication cue class on the processortherewith. The method may include the author disposing a graphicalprocessor communication cue avatar class on the processor, and disposinga processor communication cue class associated with the processorcommunication cue avatar class on the processor such that instantiatinga processor communication cue avatar from the processor communicationcue avatar class instantiates the processor communication cue from theprocessor communication cue class on the processor therewith. The methodmay include the author disposing a graphical processing cue avatar classon the processor, and disposing a processing cue class associated withthe processing cue avatar class on the processor such that instantiatinga processing cue avatar from the processing cue avatar classinstantiates the processing cue from the processing cue class on theprocessor therewith. The method may include the author disposing agraphical storage cue avatar class on the processor, and disposing astorage cue class associated with the storage cue avatar class on theprocessor such that instantiating a storage cue avatar from the storagecue avatar class instantiates the storage cue from the storage cue classon the processor therewith. The method may include the author disposinga graphical retrieval cue avatar class on the processor, and disposing aretrieval cue class associated with the retrieval cue avatar class onthe processor such that instantiating a retrieval cue avatar from theretrieval cue avatar class instantiates the retrieval cue from theretrieval cue class on the processor therewith.

The author may configure the steps and links such that the script isnon-linear. The author may engage at least one of the steps with aplurality of the links to a plurality of the subsequent steps. Theauthor may assign a plurality of the cues to at least one the step.

The graphical interface may include a drag-and-drop interface. The stepavatars may be adapted to be dragged-and-dropped onto a script space ofthe graphical interface to configure the steps for the script. The linkavatars may be adapted to be dragged-and-dropped onto a script space toconfigure the links for the script. The link avatars may be adapted tobe drawn among the step avatars to configure the links for the script.The output cue avatars may be adapted to be dragged-and-dropped onto thestep avatars to associate the output cues with the steps, the input cueavatars may be adapted to be dragged-and-dropped onto the step avatarsto associate the input cues with the steps.

The output cue avatars may include a menu and/or a command line withinthe step adapted to configure the output cues associated with the steps,and the input cue avatars may include a menu and/or a command linewithin the step adapted to configure the input cues associated with thesteps.

In another embodiment, an apparatus is provided that includes a bodyadapted to be worn on a head of an author, a processor engaged with thebody, a stereo graphical display in communication with the processor anddisposed on the body such that when the body is worn by the author thegraphical display is facing and proximate eyes of the author, a depthcamera in communication with the processor and disposed on the body suchthat when the body is worn by the author the depth camera faces at leastsubstantially forward with respect to the head of the author, a colorcamera in communication with the processor and disposed on the body suchthat when the body is worn by the author the color camera faces at leastsubstantially forward with respect to the head of the author, and agraphical interface disposed on the processor. The graphical interfaceincludes a graphical step avatar class disposed on the processor and astep class associated with the step avatar class disposed on theprocessor such that instantiating a step avatar from the step avatarclass instantiates a step from the step class on the processortherewith, a graphical link avatar class disposed on the processor and alink class associated with the link avatar class disposed on theprocessor such that instantiating a link avatar from the link avatarclass instantiates a link from the link class on the processortherewith, a graphical output cue avatar class disposed on the processorand an output cue class associated with the output cue avatar classdisposed on the processor such that instantiating an output cue avatarfrom the output cue avatar class instantiates an output cue from theoutput cue class on the processor therewith, and a graphical input cueavatar class disposed on the processor and an input cue class associatedwith the input cue avatar class disposed on the processor such thatinstantiating an input cue avatar from the input cue avatar classinstantiates an input cue from the input cue class on the processortherewith.

Each step is adapted to correspond with an event of a procedure. Eachlink is adapted to correspond with sequencing the events. The outputcues are adapted to be associated with at least one script step and tospecify a graphical output from the processor to an operator for thestep, the graphical output including guidance to the operator forexecuting the event of the procedure associated with the step. The inputcues are adapted to be associated with at least one script step and tospecify a graphical input from the operator to the processor for thestep, the input providing feedback regarding the operator executing theevent of the procedure associated with the step. The steps, the links,the input cues, and the output cues cooperate to form a script, thescript being at least partially operator-implemented, and executing thescript substantially corresponds to executing the procedure.

In another embodiment, an apparatus is provided that includes aprocessor, an outputter in communication with the processor, an inputterin communication with the processor, and a graphical interface disposedon the processor. The graphical interface includes a graphical stepavatar class disposed on the processor and a step class associated withthe step avatar class disposed on the processor such that instantiatinga step avatar from the step avatar class instantiates a step from thestep class on the processor therewith, a graphical link avatar classdisposed on the processor and a link class associated with the linkavatar class disposed on the processor such that instantiating a linkavatar from the link avatar class instantiates a link from the linkclass on the processor therewith, a graphical output cue avatar classdisposed on the processor and an output cue class associated with theoutput cue avatar class disposed on the processor such thatinstantiating an output cue avatar from the output cue avatar classinstantiates an output cue from the output cue class on the processortherewith, and a graphical input cue avatar class disposed on theprocessor and an input cue class associated with the input cue avatarclass disposed on the processor such that instantiating an input cueavatar from the input cue avatar class instantiates an input cue fromthe input cue class on the processor therewith.

Each step is adapted to correspond with an event of a procedure, andeach link is adapted to correspond with sequencing the events. Theoutput cues are adapted to be associated with at least one script stepand to specify a graphical output from the processor to an operator forthe step, the graphical output including guidance to the operator forexecuting the event of the procedure associated with the step. The inputcues are adapted to be associated with at least one script step and tospecify a graphical input from the operator to the processor for thestep, the input providing feedback regarding the operator executing theevent of the procedure associated with the step. The steps, the links,the input cues, and the output cues are adapted to cooperate to form ascript, the script being at least partially operator-implemented.Executing the script substantially corresponds to executing theprocedure.

The processor may be disposed within a head-mounted display, a desktopcomputer, a laptop computer, a tablet computer, a smartphone, and/or aportable electronic device.

In another embodiment, a method is provided that includes means forauthoring a script for a procedure onto a processor of a portableelectronic device, the procedure being at least partiallyoperator-implemented. The script includes steps corresponding to eventsof the procedure, links corresponding with sequencing the events witheach step except a final step being engaged with a subsequent step by atleast one link, and cues with each the cue being associated with atleast one of the steps and each the step having at least one the cueassociated therewith. The cues include output cues specifying outputfrom the processor, at least one output cue specifying the output beingto the operator, the output to the operator including guidance to theoperator for executing the procedure. The cues also include input cuesspecifying input to the processor, at least one input cue specifying theinput being from the operator, the input from the operator providingfeedback regarding the operator executing the procedure.

The means for authoring the script include means for disposing agraphical step avatar class on the processor and disposing a step classassociated with the step avatar class on the processor such thatinstantiating a step avatar from the step avatar class instantiates thestep from the step class on the processor therewith, means for disposinga graphical link avatar class on the processor and disposing a linkclass associated with the link avatar class on the processor such thatinstantiating a link avatar from the link avatar class instantiates thelink from the link class on the processor therewith, means for disposinga graphical output cue avatar class on the processor and disposing anoutput cue class associated with the output cue avatar class on theprocessor such that instantiating an output cue avatar from the outputcue avatar class instantiates the output cue from the output cue classon the processor therewith, and means for disposing a graphical inputcue avatar class on the processor and disposing an input cue classassociated with the input cue avatar class on the processor such thatinstantiating an input cue avatar from the input cue avatar classinstantiates the input cue from the input cue class on the processortherewith. The means for authoring the script also include means forinstantiating step avatars in a graphical interface for the processor soas to instantiate steps on the processor, means for instantiating linkavatars in the graphical interface so as to instantiated links on theprocessor, and means for instantiating cue avatars in the graphicalinterface for the processor so as to instantiate cues on the processor.Executing the script at least substantially corresponds to executing theprocedure.

In another embodiment, a method is provided that includes enablingauthoring of a script for a procedure onto a processor of a portableelectronic device. The script includes script steps corresponding toevents of the procedure, and script links corresponding to sequencingthe events with each step except a final step being engaged with asubsequent step by at least one link. Each step includes an output cueassociated with the script step and specifying a graphical output fromthe processor to the operator for the step, the graphical outputincluding guidance to the operator for executing the event of theprocedure associated with the step, and an input cue associated with thescript step and specifying a graphical input from the operator to theprocessor for the step, the input providing feedback regarding theoperator executing the event of the procedure associated with the step.

Enabling authoring of the script includes an architect defining agraphical step avatar class adapted to be disposed on the processor,defining a step class adapted to be disposed on the processor, andassociating the step class with the step avatar class such thatinstantiating a step avatar from the step avatar class instantiates thestep from the step class on the processor therewith. Enabling authoringof the script includes the architect defining a graphical link avatarclass adapted to be disposed on the processor, defining a link classadapted to be disposed on the processor, and associating the link classwith the link avatar class such that instantiating a link avatar fromthe link avatar class instantiates the link from the link class on theprocessor therewith. Enabling authoring of the script includes thearchitect defining a graphical output cue avatar class adapted to bedisposed on the processor, defining an output cue class adapted to bedisposed on the processor, and associating the output cue class with theoutput cue avatar class such that instantiating an output cue avatarfrom the output cue avatar class instantiates the output cue from theoutput cue class on the processor therewith. Enabling authoring of thescript includes the architect defining a graphical input cue avatarclass on the processor, defining an input cue class associated with theinput cue avatar class, and associating the input cue class with theinput cue avatar class such that instantiating an input cue avatar fromthe input cue avatar class instantiates the input cue from the input cueclass on the processor therewith.

Enabling authoring of the script includes the architect defining agraphical interface for the processor adapted to enable an author toinstantiate step avatars in the graphical interface for the processor soas to instantiate steps on the processor, to instantiate link avatars inthe graphical interface so as to instantiate links on the processor, andto instantiate cue avatars in the graphical interface for the processorso as to instantiate cues on the processor. Executing the scriptsubstantially corresponds to executing the procedure.

In another embodiment, a method is provided that includes enablingauthoring of a script for a procedure onto a processor of a portableelectronic device, the procedure being at least partiallyoperator-implemented. The script includes steps corresponding withevents of the procedure, links corresponding with sequencing the eventswith each step except a final step being engaged with a subsequent stepby at least one link, and cues with each cue being associated with atleast one step and each step having at least one cue associatedtherewith. The cues include output cues specifying output from theprocessor, at least one output cue specifying the output being to theoperator, the output to the operator including guidance to the operatorfor executing the procedure. The cues also include input cues specifyingan input to the processor, at least one input cue specifying the inputbeing from the operator, the input from the operator providing feedbackregarding the operator executing the procedure.

Enabling authoring of the script includes an architect defining agraphical step avatar class adapted to be disposed on the processor,defining a step class adapted to be disposed on the processor, andassociating the step class with the step avatar class such thatinstantiating a step avatar from the step avatar class instantiates thestep from the step class on the processor therewith. Enabling authoringof the script includes the architect defining a graphical link avatarclass adapted to be disposed on the processor, defining a link classadapted to be disposed on the processor, and associating the link classwith the link avatar class such that instantiating a link avatar fromthe link avatar class instantiates the link from the link class on theprocessor therewith. Enabling authoring of the script includes thearchitect defining a graphical output cue avatar class adapted to bedisposed on the processor, defining an output cue class adapted to bedisposed on the processor, and associating the output cue class with theoutput cue avatar class such that instantiating an output cue avatarfrom the output cue avatar class instantiates the output cue from theoutput cue class on the processor therewith. Enabling authoring of thescript includes the architect defining a graphical input cue avatarclass on the processor, defining an input cue class associated with theinput cue avatar class, and associating the input cue class with theinput cue avatar class such that instantiating an input cue avatar fromthe input cue avatar class instantiates the input cue from the input cueclass on the processor therewith. Enabling authoring of the scriptincludes the architect defining a graphical interface for the processoradapted to enable an author to instantiate step avatars in a graphicalinterface for the processor so as to instantiate steps on the processor,to instantiate link avatars in the graphical interface so as toinstantiate links on the processor, to instantiate cue avatars in thegraphical interface for the processor so as to instantiate cues on theprocessor. Executing the script at least substantially corresponds toexecuting the procedure.

The architect may define the output cue class such that the output cuesare specifiable by the author to include text, a schematic, a graphic,an image, an animation, a video, audio, augmented reality content,virtual reality content, stereo content, an executable instruction,and/or an external communication.

The architect may define the output cue class such that the output cuesare specifiable by the author to execute automatically. The architectmay define the output cue class such that the output cues arespecifiable by the author as executing responsive to a timer, a usercommand, and/or an environmental factor. The architect may define theoutput cue class such that the output cues are specifiable by the authoras directing the output cues to a recipient other than the operator.

The architect may define the output cue class such that the output cuesare specifiable by the author that satisfying the output cue includesdelivering the output from the processor. The architect may define theoutput cue class such that the output cues are specifiable by the authorthat confirming delivery of the output satisfies the output cue.

The architect may define the input cue class such that the input cuesare specifiable by the author to include text, an image, a video, audio,augmented reality content, virtual reality content, stereo content, anexecutable instruction, an external communication, a selection input, amachine-readable code, and/or sensor data.

The architect may define the input cue class such that the input cuesare specifiable by the author to execute automatically. The architectmay define the input cue class such that the input cues are specifiableby the author as executing responsive to a timer, a user command, and/oran environmental factor.

The architect may define the input cue class such that the input cuesare specifiable by the author that satisfying the input cue includesreceiving the input to the processor. The architect may define the inputcue class such that the input cues are specifiable by the author thatsatisfying the input cue includes confirming a receipt of the input,confirming a presence of content in the input, confirming a type of thecontent in the input, confirming a format of the content in the input,confirming the content in the input satisfies an input standard, and/orconfirming a presence of a feature in the content in the input.

Enabling authoring the script may include the architect defining agraphical identity cue avatar class adapted to be disposed on theprocessor and an identity cue class adapted to be disposed on theprocessor and associating the identity cue class with the identity cueavatar class such that instantiating an identity cue avatar from theidentity cue avatar class instantiates the identity cue from theidentity cue class on the processor therewith, defining a graphicaloperator communication cue avatar class adapted to be disposed on theprocessor and an operator communication cue class adapted to be disposedon the processor and associating the operator communication cue classwith the operator communication cue avatar class such that instantiatingan operator communication cue avatar from the operator communication cueavatar class instantiates the operator communication cue from theoperator communication cue class on the processor therewith, defining agraphical processor communication cue avatar class adapted to bedisposed on the processor and a processor communication cue classadapted to be disposed on the processor and associating the processorcommunication cue class with the processor communication cue avatarclass such that instantiating a processor communication cue avatar fromthe processor communication cue avatar class instantiates the processorcommunication cue from the processor communication cue class on theprocessor therewith, defining a graphical processing cue avatar classadapted to be disposed on the processor and a processing cue classadapted to be disposed on the processor and associating the processingcue class with the processing cue avatar class such that instantiating aprocessing cue avatar from the processing cue avatar class instantiatesthe processing cue from the processing cue class on the processortherewith, defining a graphical storage cue avatar class adapted to bedisposed on the processor and a storage cue class adapted to be disposedon the processor and associating the storage cue class with the storagecue avatar class such that instantiating an storage cue avatar from thestorage cue avatar class instantiates the storage cue from the storagecue class on the processor therewith, and defining a graphical retrievalcue avatar class adapted to be disposed on the processor and a retrievalcue class adapted to be disposed on the processor and associating theretrieval cue class with the retrieval cue avatar class such thatinstantiating a retrieval cue avatar from the retrieval cue avatar classinstantiates the retrieval cue from the retrieval cue class on theprocessor therewith.

The architect may define the step class and the link class so as toenable the script to be authored non-linearly. The architect may definethe step class and the link class so as to enable the steps to engagewith a plurality of links to a plurality of subsequent steps. Thearchitect may define the step class so as to enable the steps toaccommodate a plurality of the cues therein.

The architect may define the graphical interface as a drag-and-dropinterface. The architect may define the step avatars so as to enable theauthor to drag-and-drop the step avatars onto a script space of thegraphical interface to configure the steps for the script. The architectmay define the link avatars so as to enable the author to drag-and-dropthe link avatars onto a script space to configure the links for thescript. The architect may define the link avatars so as to enable theauthor to draw the link avatars among the step avatars to configure thelinks for the script. The architect may define the output cue avatars soas to enable the author to drag-and-drop the output cue avatars onto thestep avatars to associate the output cues with the steps, and may definethe input cue avatars so as to enable the author to drag-and-drop theinput cue avatars onto the step avatars to associate the input cues withthe steps.

The architect may define the output cue avatars so as to include atleast one of a menu and a command line to enable the author to associatethe output cues with the steps, and may define the input cue avatars soas to include at least one of a menu and a command line to enable theauthor to associate the input cues with the steps.

In another embodiment, an apparatus is provided that includes means forenabling authoring of a script for a procedure onto a processor of aportable electronic device, the procedure being at least partiallyoperator-implemented. The script includes steps corresponding withevents of the procedure, links corresponding with sequencing the eventswith each step except a final step being engaged with a subsequent stepby at least one link, and cues with each cue being associated with atleast one step and each step having at least one cue associatedtherewith. The cues include output cues specifying an output from theprocessor, at least one output cue specifying the output being to theoperator, the output to the operator including guidance to the operatorfor executing the procedure. The cues also include input cues specifyingan input to the processor, at least one input cue specifying the inputbeing from the operator, the input from the operator providing feedbackregarding the operator executing the procedure.

The means for enabling authoring of the script include means fordefining a graphical step avatar class adapted to be disposed on theprocessor, means for defining a step class adapted to be disposed on theprocessor and associating the step class with the step avatar class suchthat instantiating a step avatar from the step avatar class instantiatesthe step from the step class on the processor therewith, means fordefining a graphical link avatar class adapted to be disposed on theprocessor, means for defining a link class adapted to be disposed on theprocessor and associating the link class with the link avatar class suchthat instantiating a link avatar from the link avatar class instantiatesthe link from the link class on the processor therewith, means fordefining a graphical output cue avatar class adapted to be disposed onthe processor, means for defining an output cue class adapted to bedisposed on the processor and associating the output cue class with theoutput cue avatar class such that instantiating an output cue avatarfrom the output cue avatar class instantiates the output cue from theoutput cue class on the processor therewith, means for defining agraphical input cue avatar class on the processor, and means fordefining an input cue class associated with the input cue avatar classand associating the input cue class with the input cue avatar class suchthat instantiating an input cue avatar from the input cue avatar classinstantiates the input cue from the input cue class on the processortherewith.

The apparatus includes means for defining a graphical interface for theprocessor adapted to enable an author to instantiate step avatars in agraphical interface for the processor so as to instantiate steps on theprocessor, to instantiate link avatars in the graphical interface so asto instantiate links on the processor, to instantiate cue avatars in thegraphical interface for the processor so as to instantiate cues on theprocessor. Executing the script at least substantially corresponds toexecuting the procedure.

FIG. 1 shows a flowchart for confirming the execution of a procedurewith regard to a script operator, according to an embodiment. As notedpreviously, it may be useful to consider certain such arrangements asbeing or at least resembling “active flow charts”. That is, a script maybe visualized as a flowchart, with individual steps as being visualizedas blocks within the flow chart, links as arrows, cues as instructionsetc. for/within blocks representing steps, etc. However, where aconventional flowchart may be static, e.g. a printed document thatmerely “sits on the page”, a script as described herein may be active inthat the script may cause the output to be delivered, may accept input,may interact with a human operator, adapt to varying circumstances, etc.Moreover, as addressed in more detail subsequently, scripts such asdescribed herein may be active in that the script may be readily createdand modified, e.g. by “drag and drop” manipulation of steps within thescript, etc.

However, it is emphasized that while the notion of an active flowchartmay be illustrative, the script may not be or appear as a flow chart inactual fact (though such also is not prohibited). Thus, for an operatorfollowing the script, e.g. receiving output for certain steps,performing steps, providing input for certain steps, etc., the scriptmay not visibly be presented as a flowchart, and indeed the scriptitself may not appear at all (even though output may be displayed,etc.). Thus while the notion of scripts as active flow charts may beuseful in understanding certain embodiments and/or applications, itshould be understood that such a notion is not limiting.

In FIG. 1 , a script is instantiated 102 onto the processor of aportable electronic device. Typically though not necessarily, theprocessor may be a digital electronic processor, and the portableelectronic device a head-mounted display, smartphone, tablet, laptopcomputer, etc. However, the use of a portable electronic device is anexample only; so long as some capability exists to execute the steps asshown and described, substantially any device or system may be suitable.Likewise, although for purposes of explanation the processor may beconsidered to be a digital electronic processor adapted to executeexecutable instructions, the particulars of the processor are notlimiting unless specifically noted herein.

Similarly, the manner by which the script is instantiated is notlimited. Typically though not necessarily, the script may beinstantiated onto the processor from a data store such as a hard drive,solid-state drive, cloud storage, etc., though other arrangements alsomay be suitable.

In instantiating 102 the script, several features may be instantiatedwith and/or as part of the script.

Script steps are instantiated 102A on the processor as part of thescript. As noted previously, script steps (sometimes referred to hereinsimply as “steps”) typically address some action, such as may correspondwith an event in a procedure. For example, a script step may refer toevents such as identifying a machine, opening or manipulating someportion of the machine, inspecting or modifying some component of themachine, etc. (though other arrangements may be suitable). In theaforementioned analogy of an active flow chart, script steps may beconsidered as the various “blocks” in such a flow chart.

Script links are instantiated 102B on the processor as part of thescript. As also noted previously, script links facilitate sequencingamong steps within a script, e.g. a link from step A to step B such thatstep B follows after step A. For simplicity links may be described incertain instances as being linear and unitary, that is, each step has alink from one previous step to one following step. However, multiplelinks may lead to and/or proceed from any given step, and thus links donot necessarily merely produce a linear sequence of steps.

Still with reference to FIG. 1 , in the example arrangement thereinoutput cues are instantiated 102C on the processor as part of thescript. Output cues specify outputs from the script to some target;typically though not necessarily the output cue specifies information, acommand, etc. sent from the processor to a human operator or to a deviceor system other than the processor. For example, an output cue mightdirect output of a video or an image to an operator via a graphicaldisplay, might direct output of a command to a camera to capture animage, etc. It is noted that the output cue is not necessarilyequivalent to the output; the output cue specifies that output is to besent, but does not necessarily include the output itself (though suchalso is not prohibited). For example, images, videos, schematics, etc.may be accessed from some source other than the script itself, such as ahard drive, solid-state drive, cloud storage, etc.

Thus, a given output cue may provide guidance, information, etc. to anoperator. It is noted that output cues are not limited only to providingoperator guidance; output cues may also provide guidance and/or controlto other systems (e.g. sending a command to a machine to shut down formaintenance, in addition to or in place of advising the operator toensure that the machine is shut down), may specify output to thirdparties, etc.

Input cues also are instantiated 102D on the processor as part of thescript. Input cues specify inputs to be received from some source to thescript. Typically though not necessarily the input cue acceptsinformation, a command, etc. from the operator or some device or systemother than the processor sent to the processor. For example, an inputcue may receive an image from a camera, may receive voice or gestureinput, may accept data communicated from an external instrument (such asan operating data file from a machine undergoing maintenance), etc. Asnoted with regard to output cues, input cues are not necessarilyequivalent to input.

With regard to terminology, it is noted that cues typically are referredto herein as being “output” or “input” with regard to the script itselfIn certain embodiments, this may be equivalent or at least similar tobeing outputs from the processor, or inputs to the processor, in thatthe script may be instantiated on the processor. That is, if the script“lives” on the processor, communicating with the processor may equate tocommunicating with the script.

Continuing in FIG. 1 , a step in the script is executed 104. Typicallythough not necessarily the script itself may identify a starting step.In other instances the operator may specify an initial step, somealgorithm may be in place (e.g. as part of the script itself) todetermine which step should be executed 104 first, etc. How the initialstep is selected is not limited.

In executing 104 the script step in question, an output cue associatedwith the relevant step is executed 104A. Thus, whatever output isspecified by the output cue is directed to whatever target is specifiedby the output cue, in whatever manner is specified by the output cue,etc. For example, the output cue may specify delivering a graphicalrendition of a schematic to the operator via the screens of a headmounted display (e.g. one in which the processor is disposed), etc. Anoutput cue may be very simple (e.g. display a single word or symbol on ascreen, send a simple command to some device, deliver an error beep viaan audio speaker), etc. However, output cues may also be very complex,and the complexity (or lack of same) of output cues is not limited. Tocontinue the example of a schematic that is to be displayed, an outputcue may for example specify that the schematic is to be delivered inmono or stereo, in color or black and white, etc., may utilize a querycommand for determining the optimum settings for the available display,may include an algorithm for determining what size or contrast settingsto use for the display, etc.

An input cue associated with the relevant step also is executed 104B.Whatever input specified by the input cue is accepted from whateversource is specified by the input cue, etc. For example, the input cuemay specify receiving information from an RFID tag on a machine that isto be serviced (possibly in response to an output cue specifying thatthe RFID tag be queried for that information). It is noted that such“machine to machine” inputs and associated input cues may varyconsiderably (in addition to variations in other types of inputs andinput cues). For example, as noted RFID tags may be read; in addition,or instead, bar codes may be read, Bluetooth systems may beinterrogated, wifi communication may be established, etc. Output cueslikewise may be machine-to-machine and may vary in a similarly broaddegree. Thus, although not limited thereto, it should be understood thatmachine to machine inputs and outputs may address substantially anymachine capable of communicating in some manner (even if only passively,such as a barcode); consequently, many members and potentially anymember of the “internet of things” may be a recipient of output and/or aprovider of input.

As with output cues, input cues may be simple or complex. For example,the input cue may specify only that data must be received, may specifythe format of data, may evaluate the general configuration of data (e.g.if a machine ID is known to be a six-digit numerical string, the inputcue may compare the data received against some standard to verify thatindeed the data is a six-digit numerical string), may specify the sourceof the input (e.g. verifying in some manner that the data is in factfrom the correct RFID tag), may consider the data itself (e.g. is themachine ID the correct ID), etc.

It is noted that for simplicity it is assumed with regard to FIG. 1 thateach step has exactly one output step and one input step. In practice, ascript step may have no output cues, one output cue, or multiple outputcues associated therewith, and likewise may have no input cues, oneinput cue, or multiple input cues associated therewith. As described inmore detail subsequently herein, cues other than input and output cuesalso may be present, in addition to or in place of input and/or outputcues.

With regard to “execution” of a cue, typically though not necessarilyexecuting the cue is necessary and/or sufficient to satisfy the cue.That is, once whatever action is associated with a given cue is carriedout—for example delivering a text message to a display, for an outputcue—that cue may be considered to be satisfied. However, cues mayinclude other requirements, standards, etc. beyond simple execution(though for simplicity in certain examples herein execution andsatisfaction of a cue may be referred to as if equivalent or at leastsimilar).

For example, an output cue may require not only delivering an output butdelivering that output in a particular manner, at a particular time, toa particular device, in a particular format, to a particular operator,etc. (and likewise with input and/or other cues). In addition orinstead, cues also may specify execution with particular “triggers”,such as executing automatically, executing based on a manual input froman operator, etc.

Cues may require a response, an evaluation, additional actions, etc. Forexample, an output cue delivering a cautionary message to the operatormay require, to be satisfied, that the operator then submits someindication that he or she has seen the cautionary message. Depending onconfiguration, the response may be a cue in itself (e.g. an input cue),or the response may be part of the output cue itself (that is, theoutput is not considered “properly delivered” until and unless the userverifies receipt). As another example, an input cue accepting a digitalphotograph may be satisfied by mere receipt of a file, but may alsorequire and/or perform a verification that the file is an image file(e.g. by identifying the file extension as .jpg, .gif, .png, etc.), mayrequire and/or perform an evaluation of the image itself to verify thatthe subject matter of interest is present (e.g. through objectrecognition, by reading a bar code or QR code visible in the image), andso forth.

Cues may specify various conditions as well, such as environmentalconditions, e.g. if it is raining, delivering a particular output cuereminding an operator that a component must be kept dry. In suchinstances, if the cue is not executed in the proper manner, thenexecution in itself may not be possible and/or may not satisfy the cue.For example, an output cue that specifies delivery to a stereo displaymay not be satisfied if no stereo display is available.

However, it may also be suitable to configure cues and/or theirspecifications so that cues may be satisfied even if not fully executed.To return to the example of a cue reminding an operator to keep acomponent dry, the cue may be configured so as to be satisfied evenwithout executing (i.e. without delivering the reminder) so long asthere is no rain. Alternately, cues may be simply bypassed withoutexecution.

Thus, although in simple terms it may be convenient to refer to cues asexecuting, as though the cues consistently are carried out in full, cuesmay nevertheless be satisfied even if executed only in part or not atall. Moreover, in certain embodiments a script may be configured so thatno possible case calls for all cues to be executed; that is, if there isan output cue for a message to be delivered to the operator during rain,and an alternate output cue for a message to be delivered when there isno rain, then there may be no instance when both cues are executed for asingle iteration of a procedure (i.e. it cannot simultaneously both beraining and not be raining, thus at least one of the output cues willnot be executed).

Still with reference to FIG. 1 , in executing the step 104 a link fromthe step being executed then is followed 104C to the next step. Theexample method in FIG. 1 then continues with the step that has beenlinked to, thus executing 104 a subsequent step. Typically the methodmay proceed until either an explicit termination (a “stopping point” forthe script) is reached, until no outgoing links are available, etc.

It is noted that the ordering of executing output cue 102A and input cue102B may vary. In the example arrangement shown, an output cue is firstexecuted 102A, then an input cue 102B. However, other arrangements maybe equally suitable, and in general, it is not either logically orpractically required that output cues must precede input cues.Typically, following 104C a link to the next step is to be executed onceall appropriate cues have been executed; however, it is noted that incertain instances cues may be ignored. Indeed, cues themselves may beconditional. For example, an input cue may be configured such that if anRFID query is sent as an output cue, but no RFID response is received asan input cue, after a certain amount of time the output cue is executedagain, or the input cue accepting the RFID response is skipped and thenext cue in line (if any) is executed despite the lack of RFID response,etc. Likewise, output or input cues may exist that only execute undercertain circumstances, e.g. an output cue advising an operator toimmediately cease work if an unsafe condition is detected. Similarly,entire steps or groups of steps (and or links among those steps) mayexecute only under certain circumstances, for example, a procedure forinspecting a machine for wear may include one or more series of stepsfor replacing worn parts that only being executed if the parts beinginspected are found to be excessively worn.

Thus, not all cues necessarily will be executed, nor will all stepsnecessarily be executed, nor will all links necessarily be followed, inany given use of a script or for any given script overall.

FIGS. 2A and 2B therein are shown an example arrangement of a script 200in flow chart form. It is again emphasized that the flowchart form ofthe script 200 in FIG. 2A and FIG. 2B is illustrative; the script 200may not actually have the form of a flowchart, nor is such required,however illustrating the script 200 as if the script 200 is a flowchartmay be useful. Although as noted scripts may be non-linear and/orconditional, for clarity the arrangement in FIG. 2A and FIG. 2B islinear and has no contingencies (i.e. all steps and cues are presumed tobe executed). In addition, it is noted that the script 200 in FIG. 2Aand FIG. 2B parallels a procedure that is hypothetical; for examplepurposes the script 200 is presented as corresponding with a wearinspection of some machine, but the script 200 is not intended tonecessarily correspond with any real-world machine or inspectionprocedure.

FIG. 2A shows a script 200 in the form of a flowchart for the arrangingof a script for guiding a procedure and confirming execution, accordingto an embodiment.

(The titles/descriptions of steps, cues, etc. in FIG. 2A and FIG. 2B andin certain examples elsewhere herein are illustrative only; steps and/orcues may or may not have titles, and titles may or may not have anyparticular form.)

As may be seen, step 212 has one cue associated therewith, output cue212A, illustrated as a “folded corner” rectangle disposed within step212. Although the output cue 212A is shown as being within the step 212,this is illustrative; it is not prohibited for cues to be literally (oreven figuratively) within or part of a step, but cues also are notrequired to be part of a step. The output cue 212A specifies output toan operator, in the form of a text message: “Wear inspection for machineID [###,###].” (In the example shown, the numeral signs may represent asix-digit numerical ID for the machine in question).

Thus, in executing the step 212 to address the procedure, the output cue212A would be executed, and in executing the output cue 212A the text“Wear inspection for machine ID [###,###]” would be displayed to theoperator (e.g. through a graphical display on a smartphone, head-mounteddisplay, etc.). In more colloquial terms, step 212 may be considered astelling the operator that they are to begin performing a wear inspectionprocedure on a particular machine.

With step 212 complete, the script proceeds along a link 213 from step212 to step 214 for confirming the machine. This step 214 has three cuesassociated therewith, 214A, 214B, and 214C. As may be observed, thethree cues 214A, 214B, and 214C are connected by arrows; these are not“links” in the sense of linking steps, but rather are presented to showan order of execution of the various cues 214A, 214B, and 214C. It maynot be required in all cases to specify an order of execution, and theorder may vary (and as noted above some cues may not execute at all inat least certain instances), however for clarity a simple linear orderfor cue execution is indicated in FIG. 2A and FIG. 2B.

Cue 214A is an output cue that specifies providing output to an RFID tagthat is present (or at least presumed to be present) on the machinebeing inspected. The output for cue 214A is a query for the machine IDnumber. Cue 214B is an input cue that specifies accepting input from theRFID tag, with the expected input being the machine ID number asrequested by cue 214A. (To distinguish visually between input and outputcues, input cue 214B is shown as a folded corner rectangle like cues214A and 214C, but with the fold on the lower left; by contrast outputcues 214A and 214C show the fold on the lower right corner.) Cue 214C isanother output cue, but to the operator, specifying text output:“Confirmed: machine ID is [###,###].” (As noted in practice cues may notnecessarily be a simple linear progression; in some embodiments, a cuesimilar to 214C may exist with a message indicating that the operator isaddressing the wrong machine, etc.)

Moving on in FIG. 2A, the script proceeds along link 215 from step 214to step 216 for accessing the component that is to be inspected. One cue216A is associated with step 216. The cue 216A is an output cue thatspecifies the output of text to an operator: “Remove bolts and openaccess panel.” Once step 216 is executed, the script 200 continues alonglink 217 from step 216 to step 218.

Step 218 addresses removing the component that is to be inspected. Step218 has two cues 218A and 218B associated therewith as shown in FIG. 2A.Cue 218A is an output cue to the operator specifying text output (e.g.via a display): “Turn release lever counterclockwise to remove thecomponent.” Cue 218B is also an output cue to the operator (again e.g.via a display), showing a video of the release lever being turned andthe component being removed. As noted previously the ordering of cues isnot necessarily limited to what is shown herein, or to any particulararrangement. An example may be seen in step 218, in that although cue218B is shown to follow cue 218A, it may be equally suitable for cue218A to follow cue 218B, or for cues 218A and 218B to be executedtogether, etc. That is, it may be equally suitable to output the textinstruction before the video (as shown), or the video before the textinstruction, or the text instruction and the video at once.

FIG. 2B shows another flowchart for the arranging of a script forguiding a procedure and confirming execution, according to anembodiment. Moving from FIG. 2A to FIG. 2B, a link 219 leads from step218 in FIG. 2A to step 220 in FIG. 2B (the link 219 being shown in bothFIG. 2A and FIG. 2B).

Step 220 refers to examining the component (removed previously in step218 in FIG. 2A). Three cues 220A, 220B, and 220C are associated withstep 220. Cue 220A is an output cue to the operator specifying textoutput: “Measure wear strip with a caliper.” Cue 220B is also an outputcue to the operator, specifying the output of an image showing theproper caliper positioning for measuring the wear strip. Cue 220C is aninput cue specifying an expected voice input from the operator (e.g. viaa microphone), giving the measured value of the wear strip as determinedby the caliper.

Typically it may be anticipated that cues 220A, 220B, and 220C may beexecuted in that order, as shown by the arrows in FIG. 2B. However, itmay be understood that even when some implicit ordering of cues may bepresent (e.g. the input for the caliper measurement may be anticipatedas taking place after the instruction to perform the calipermeasurement), as noted certain cues may be ignored, sequencing may bechanged, etc. For example, if the operator were sufficiently familiarwith the procedure corresponding with the script 200, the operator maymeasure the wear strip and speak the caliper measurement into amicrophone (specified by cue 220C) before the text and image aredelivered to the operator (220A and 220B). In such case cues 220A and220B may be executed anyway, or cues 220A and 220B may be skipped, etc.However, such adaptability is not required, and certain embodiments mayeven exclude such adaptability by design; for a critical procedure, itmay be desirable (e.g. for safety purposes) to configure a script suchthat each step, each link, each cue, etc. are carried out preciselyscripted. In such case, delivering an input specified by an input cuebefore a preceding output cue has been executed (or similar) may promptan error response, revert the procedure to an earlier cue or step, etc.

Still with reference to FIG. 2B, once step 220 is executed a link 221 isfollowed from step 220 to step 222. Step 222 references replacing thecomponent in the machine following the measurement (in step 220). Step222 in some sense a reverse of step 218, and the cues 220A and 220B aresimilar to those in step 218. Cue 222A is an output cue to the operatorspecifying text output: “Reinsert component and turn release leverclockwise to secure.” Cue 222B is an output cue to the operator showinga video of the component being re-inserted and the release lever beingturned.

With step 222 executed, a link 223 is followed to step 224. Step 224addresses confirming that the component has been properly re-inserted.An output cue 224A specifies output to the operator with textinstructions: “Photograph to verify component placement.” An input cue224B specifies an anticipated input of the photograph as delivered bythe operator manually activating a camera (e.g. on the portableelectronic device) to document the position of the component in themachine.

Step 224 being complete, the script 200 advances along a link 225 tostep 226. Step 226 refers to securing the machine following theinspection. An output cue 226A to the operator specifies text output:“Replace access panel and tighten bolts.” Another output cue 226Bspecifies sending a command to a camera to capture a photograph showingthat the access panel has been properly secured. An input cue 226Cspecifies receiving an expected photograph (e.g. a digital image) fromthe camera in response to the command specified in output cue 226B.

In some sense receiving a camera image documenting that access panel hasbeen replaced and bolted (as in cue 226C) may be considered to completethe procedure. Similarly, the script 200 as shown in FIG. 2A and FIG. 2Bis complete following the execution of step 226 and the cues 226A, 226B,and 226C thereof

However it should be understood that embodiments of scripts may continueafter the procedure to which those scripts correspond are nominallycomplete, and/or scripts may include steps and/or cues that do notnecessarily correspond precisely to events within the relevantprocedure. For example, automatically capturing an (as in step 226, cues226B and 226C) may or may not technically be considered part of theprocedure; the procedure may be considered to be complete when the panelis bolted, not when the panel being bolted is documentedphotographically. Likewise, subsequent analysis, data transfer, etc.related to the procedure may be part of the script and may take placeoutside the procedure itself, e.g. data collected during the proceduremay be archived by the processor into a data store, may be processedwith algorithms, may be reported to some oversight entity, etc.

Similarly, procedures may include events not directly paralleled by ascript. For example, filing a paper “sign-off” with a regulatory bodymay be considered to be part of a maintenance procedure, but may not bereflected by a script for that procedure. Thus, although a given scriptmay be intended to and may in fact substantially correspond with aparticular procedure, there may not be a perfect and/or comprehensiveone-to-one match between a procedure and a script for that procedure.

In the arrangement shown in FIG. 2A and FIG. 2B, output cues and inputcues are shown. However, other cues may be present in variousembodiments, in addition to and/or in place of output cues and inputcues. In some sense, cues may be considered collectively as being“something that's supposed to happen” in a step of a script (and/or assomething that does indeed happen in an event of a procedure to whichthe script corresponds). Indeed, output cues and input cues may betreated in some instances as similar, i.e. as “just cues”; there may belittle fundamental difference between an input cue and an output cueaside from the direction in which information and/or action flows.

However, it may be useful nevertheless to describe certain varieties ofcues. This should be understood to be illustrative, rather thanlimiting; not all cues described herein necessarily must or will bepresent in all embodiments, nor are cues other than those describedprohibited. (In addition, for further discussion below regarding thecreation of scripts, the notion of different varieties of cues also maybe useful.)

Cues may be broadly considered in terms of what the cues specify, e.g.output or input for output cues and input cues respectively.

A cue wherein the identity of some entity is determined may be referredto as an identity cue. For example, it may be desirable to determine theidentity of an operator; certain operators may be trained for aparticular procedure while others are not trained, some operators may becertified or legally licensed while others are not, some operators mayhave authorization to carry out a procedure (or even to access thescript corresponding to that procedure) while others do not, someoperators may be authorized to use the particular processor and/or adevice with the processor therein while others are not so authorized,etc.

Determining the identity of an operator may be carried out in a varietyof manners. For example, for a head-mounted display, a camera aimedtoward the wearer (e.g. for eye tracking) may image the operator's face,and face recognition then could be used to determine the identity of theoperator. Similarly, for a smartphone, a camera image of the user mightbe obtained. Other arrangements also may be suitable.

It is noted that determining “the identity” of an operator does notnecessarily mean specifying a single human being. Rather, in someembodiments, it may be sufficient to verify that a given operator is oneof a group without necessarily determining which individual within thatgroup the operator may be. Likewise, it may be sufficient to determinewhether an operator is properly equipped (e.g. an operator may beauthorized for a particular procedure only if the operator is wearingproper safety gear).

In addition, it may be desirable to determine the identity of otherentities as well. For example, identifying the processor may enabletagging records generated by the script (if any) with the serial number(for example) of the processor or device, so as to support a traceablechain of data. Identifying a device in which the processor is disposedmay enable confirmation that the device is suited for a given procedure,e.g. the device has the appropriate sensors, the device is suited for aparticular environment (rated for cold or heat, certified asintrinsically safe for use around flammable gases, etc.), and so forth.Identifying the script itself may enable verifying that the script beingused is the proper and/or most recent version for a given procedure.Other identifications, and thus other forms of identity cues, also maybe suitable.

A cue wherein communication is established between the operator and someother entity may be referred to as an operator communication cue. Sincein some sense output and input cues may be considered to addresscommunication between the operator and the processor, typically anoperator communication cue may be considered to exclude cues thataddress only communication between the operator and the processor.(However, as noted the definitions of types of cues are not rigid orotherwise limited, and while varieties, as described herein, may beillustrative other arrangements may be suitable.)

An operator communication cue may, for example, facilitate communicationbetween an operator and a colleague, a supervisor, a regulatoryindividual or agency, an expert on some relevant subject, a database ofblueprints or maintenance logs, etc. Thus, as part of a procedure and/ora script (and at least potentially being logged or otherwise notedby/within that script), an operator may discuss the state of a machinebeing serviced with a colleague who serviced the machine previously, mayconsult an expert on that machine if problems or unexpected results areencountered, etc. In some sense, the operator may by use of the scripthave an expert always “at the ready” (or an inspector for sign-off, aparts department for ordering replacement components, etc.). Multipleentities may be communicated with, e.g. as a sort of “conference call”,within the procedure itself. It is noted that such other entities—e.g.experts, supervisors, etc.—need not be physically present, nor evenclose by. If an operator is performing maintenance on (for example) anobscure engine component for which there exists a single “go to person”knowledgeable about that component, the script thus may facilitatecommunication with that knowledgeable person within the procedureitself.

A cue wherein communication is established between the processor andsome other entity may be referred to as a processor communication cue(again in at least some instances excluding the operator as that otherentity, since input and output cues may already represent suchoperator-processor communication). In some sense, nearly anycommunication within the scope of a script could be considered“processor communication”, for example in that a video conference withan expert may well be managed by the same processor managing the script.However, it may be convenient to consider processor communication cuesto address communications more directly by the processor, such asnetworking with other processors, accessing databases with theprocessor, etc.

A cue wherein data and/or executable instructions are stored may beconsidered a storage cue. Likewise, a cue wherein data and/or executableinstructions are retrieved may be considered a retrieval cue.

As noted with regard to input and output cues above in the example ofFIG. 2A and FIG. 2B, inputs and/or outputs specified by input and outputcues may include data that is of value or interest, but that may or maynot be part of the script proper. For example, images, videos, etc. thatare to be displayed to an operator as output may or may not beinstantiated as a script itself. While including such data within ascript is not prohibited (and it may be useful to include commonly useddata within the script itself), neither is it required that all or evenany output data must be part of a script. For example, while it may beuseful to have the option to output a detailed set of schematics for agiven machine, or a comprehensive maintenance log, etc., suchinformation may not be necessary or even useful every time a givenprocedure is performed, by every possible operator. As a more concreteexample, a general maintenance script for a laptop computer may includeimages showing in detail how to replace the hard drive, but such datamay not be needed when merely cleaning the cooling fans.

Similarly, input data such as photographs collected, text entered, voiceinputs, etc. as may be associated with input cues may be of sufficientinterest as to merit storage, but it may not be necessary or useful topreserve such data within the script itself

Thus, storage and retrieval cues may be part of a script, facilitatingdata management within that script. The operator may not even bedirectly aware of such cues, and there may be no external indicationthat such cues exist or are being executed. That is, an operator may seeschematics that are displayed (e.g. as specified by an output cue), butmay not be aware (and may not need to be aware) that those schematicswere accessed as specified by a retrieval cue. More generally, not allcues of any sort necessarily will have any directly visible sign ofpresence or execution to an operator; while an output cue to an operatoror an input cue from an operator may be apparent to that operator, othercues (including input or output cues not addressed to the operator) maybe “transparent” to the operator.

Indeed, a script overall, and/or the arrangement of steps, links, cues,etc. in a script, may be in at least some sense transparent to theoperator. That is, the operator may not be aware, and may not need to beaware, of what the overall configuration of the script is, i.e. whatsteps exist, which step follows which step, what cues are associatedwith each step, whether cues are unique to a step or are associated withmultiple steps, etc. Rather, the operator may simply receive outputs andsend inputs as prompted by cues for various steps, and follow throughthe procedure as the script advances via links from one step to another.The operator thus may actively carry out a procedure, receiving guidancein carrying out that procedure and providing feedback from thatprocedure, without necessarily being familiar with the whole of theprocedure. More colloquially, the operator may not know the procedure,and may not see “the big picture” (nor is such necessarily required).

A cue relating to a contextual link between some element related to ascript and some other element or elements may be referred to as acontext cue. The notion of a context cue is broad and may be somewhatnon-intuitive, thus some explanation may be in order.

In essence, a context cue is a cue that in some manner acquires and/orassociates information for a step, procedure, etc. Typically though notnecessarily, context cues may be at least partially automated.

As a more concrete example, consider an arrangement wherein an operatoris following a procedure to replace a tire on a particular model ofaircraft, at a particular airport. A context cue may be attached to ascript and/or to one or more steps in that script, such that when theoperator approaches that aircraft the specific maintenance log for thataircraft is automatically searched for (e.g. in a maintenance databasefor the airport, though this is an example only), or so that specificprocedural requirements or notes for maintenance at that airport areaccessed, etc. Similarly, the context could include known maintenanceissues, the location of tools needed, the availability of a suitablebay, etc.

It is noted that context cues may, in fact, be entirely functionalapplications unto themselves, rather than merely being data or flagsattached to a step or script. (This may be likewise true with othercues, but given the breadth and flexibility of what may be consideredthe context for various embodiments, context cues that are in factcontext “apps” may be particularly notable.) Thus, a context cue may bea link to a context app, which may for at least some embodiments receiveinputs, send outputs, process data, etc. as an entity distinct from thescript, communicating the script without being part of the script instrict terms.

Thus, context cues (and/or context apps, if a distinction is madetherebetween for a given embodiment) may address a wide variety ofcontextual factors, providing a similarly wide range of additionalfunctionality (e.g. determining where a machine or operator is,identifying the machine, acquiring information on the particularmachine, accessing any work orders on that machine and/or error codesfrom that machine, evaluating environmental conditions near the machinewith regard to effects on the procedure to be performed, etc.)

Other varieties of cues, including but not limited to flag cues (whereindata, cues, steps, etc. may be highlighted for special attention),keyword cues (wherein some keyword or other identifier is associatedwith a script, a procedure, a machine, a particular instance ofexecuting a script, etc. so as to facilitate later searching), and linkcues (wherein a given machine, procedure, script, etc. is linked toanother machine, procedure, script, etc. as being potentially relevant)also may be suitable. It is emphasized that cue varieties may be usedwhere convenient but such grouping need not be rigidly defined or evenused at all; in particular, overlap, redundancy, “fuzziness” etc. amonggroups is permissible. For example, it may be permissible for cues to beconfigured such that the same effects could be produced either by anoutput cue or a processor communication cue, etc.

What has been referred to as the “variety” of a cue may be consideredmore generally to be one of a number of possible features of cues. Thusin addition to considering cues as potentially divisible into differentvarieties for convenience, it may also be useful to consider variousother potential features of cues.

Cue variety, as noted, may be considered as to grouping cues based onsome broad sense of what the cue is “for”, e.g. for providing an output,for accepting an input, for setting up a communication channel with somethird party, for performing internal analysis, etc.

At least certain cues may be described as having an “address” feature.This is at least implicit in certain examples already described, forexample, some output cues in FIG. 2A and FIG. 2B specify that the outputis to be delivered to the operator (though without those examplesexplicitly referencing the operator as an “address”). Likewise, someinput cues in FIG. 2A and FIG. 2B specify that input is from theoperator (again without explicitly referring to the operator as the“address”).

For simplicity, the term “address” may be used regardless of whether theaddress refers to a source, a destination, or both. Furthermore, anaddress may specify a destination, a source, or both. Where one or theother is not specified, the destination or source may be implied; forexample, it may be that the source for an output cue is assumed to bethe script or the processor on which the script is disposed and thatonly the destination is explicitly specified as an address for thatoutput cue.

Different levels of specificity may be suitable with regard to address;an output cue might specify one particular operator, or might specify aparticular route for providing the output to the operator such as adisplay screen (which the operator may be presumed to be viewing).Alternately, an address for a cue may refer to the route itself withoutnecessarily referencing the destination (e.g. the operator), thus theaddress for an output cue may be (or may be considered to be) agraphical display, an audio speaker, etc.

Typically though not necessarily, the address feature for a cue may bespecified or “set” when the script is created or modified. However,specifying or changing an address in “mid-script” or “mid-procedure”also is not prohibited. For example, if a device has both a graphicaldisplay and an audio speaker, and the audio speaker is muted, theaddress for an output cue may change in mid-script from being directedto the (now-muted) speaker to being directed to the graphical display.In addition, cue features like cues, steps, etc. may be complex and/orconditional, e.g. an address feature for a cue may specify that outputis to be delivered as stereo graphics if the display supports stereooutput, but as mono graphics, if the stereo output is not supported. Itis also noted that an address feature may not be required for all cues,and/or may be implicit rather than explicit as noted above for some orall cues. Such comments also apply similarly to other cue features asdescribed herein (i.e., cue features other than address may be set orreset when a script is created or modified, may be complex and/orconditional, etc.).

Examples of a specific address feature for an output cue may include anyoperator, any of a group of operators, a specific operator, a graphicaldisplay, any available graphical display, a particular graphicaldisplay, a mono display, a stereo display, an LED display, an OLEDdisplay, a CRT display, a plasma display, a projection display, an audiospeaker, any available audio speaker, a particular audio speaker, a monospeaker, a stereo speaker, and a haptic output. Examples of a specificaddress feature for an input cue may include an instrument, a sensor, animager, a camera, a digital camera, a CCD camera, a CMOS camera, avisible light camera, an RGB camera, a near-infrared camera, a thermalcamera, a depth camera, a microphone, a touch screen, a keypad, a manualcontrol, and a gesture input pathway. Examples of a specific addressfeature for an operator communication cue may include a colleague, anexpert, a specific individual, any available individual, a communicationdevice (e.g. a smartphone, a laptop computer, a desktop computer, a headmounted display), and a destination code (e.g. an IP address, a webaddress, a telephone number, an email account). Examples of a specificaddress feature for a processor communication cue may include aprocessor, a processor input/output pathway, and a database. Examples ofa specific address feature for a processing cue may include analgorithm, a group of algorithms, a subroutine, and a specializedprocessor. Examples of a specific address for a storage cue and/or aretrieval cue may include a data store, a hard drive, a removablemedium, an optical medium, a magnetic medium, a solid state drive, and acloud storage. Such examples are not limiting, and other addressfeatures (for those cues noted and/or other cues) also may be suitable.

Cues may include an actuation feature. That is, a cue may specifywhether (for example) some or all of the cue is to be executedautomatically, manually, etc. As a more concrete example, an input cuefor receiving an image input from a camera may specify that the camerais to be actuated automatically by the processor on which the script isdisposed, actuated automatically by the camera itself, actuated manuallyby an operator physically pressing a button (or using a command word,etc.), actuated manually by some other party (such as by a remote expertin communication with the operator), and so forth.

Cues may include a permission feature, for example, so as to specifywhether certain operators, certain devices, etc. are authorized to carryout a cue, whether the cue may only be executed at certain times andplaces or under certain conditions, and so forth. As a more concreteexample, if an inspection of an electronic component includes apermission feature specifying that a grounding strap must be worn forthe inspection, the cue may not execute without confirmation that agrounding strap is being worn by the operator carrying out theinspection. (It is noted that such permissions likewise may beimplemented at a step level, a procedure level, etc.)

Cues may include environmental features. That is, certain cues mayexecute only when certain environmental factors are present, absent,within a given range, etc. For example, an input cue for receiving anRGB image may include an environment feature that specifies the cue isonly to execute when the ambient light levels are above some minimum. Asanother example, a related cue for receiving a near-infrared image mayinclude a parallel environment feature that specifies the cue is only toexecute if light levels are below that same minimum. (Such examples mayconsider that RGB cameras may function poorly in dim light, while nearinfrared may function poorly in sunlight or other bright light.)

Environmental features for cues may range very widely. Examples ofenvironmental features that may be specified for various cues includebut are not limited to the presence, absence, and/or levels of visiblelight (as noted above), other forms of light (e.g. near infrared,ultraviolet), radio waves, electromagnetic interference, toxic gases,combustible gases, combustible dusts, other flammables, ignitionsources, high voltage sources, extreme heat or cold, sharp or crushinghazards, and biological, chemical, or radiological hazards. Featuresthat may not necessarily be considered part of an “environment” incolloquial use nevertheless may be considered as environment featuresfor cues (or may be considered as other features for cues), includingbut not limited to the presence or absence of people generally orcertain groups or individuals, the time, the location, whether a device(if any) to be addressed is operating (e.g. when doing maintenance on amachine, is the machine turned off?), whether other devices areoperating in the vicinity, etc.

Cues may include conditional and/or logical features. For example,features may specify whether other features are to be considered (e.g.always considering levels of methane gas as an environmental feature forscripts anticipated for use in oil rigs, but not considering the levelof methane as an environmental feature in an agricultural setting),features may specify logical constraints (e.g. consider feature X ifconditions A and B are present, unless condition C is also present,etc.).

Preceding examples of cue features are not limiting, and other featuresmay be equally suitable. In addition, the grouping of cue features asdescribed is an example only, and other arrangements may be suitable.For example, a time feature may be specified as distinct fromenvironment features, all features could be collected into a singleomnibus category, feature categories may be fluid and/or modifiable byoperators or other persons, etc.

Furthermore, cues may also be conceptualized in terms of standards for acue, or otherwise, rather than necessarily only in terms of features ofcues. For example, an output cue may include an output standard; theoutput standard may define where the output is directed (to the user, toa display visible to the user, to some other person or system, etc.),how/when the output is to be delivered (immediately, automatically, uponuser request, after a time delay, upon detection of some condition,etc.), the format of the output (the resolution of an image, the framerate of a video, the initial orientation of a 3D model, etc.), requiredor prohibited conditions (do not deliver output when the user isotherwise occupied, only deliver stereo graphical output to a displaythat has stereo capability, etc.), and so forth. An input cue likewisemay include an input standard, wherein the input standard may definewhere the output is to be acquired (from a user, via keyboard, viagesture, via a camera that may detect a gesture, via an algorithm thatdetects gestures within images, etc.), the format of the input (anydata, audio data specifically, image data, an image with a bar codetherein, a specific bar code, etc.), required or prohibited conditions(an image must be obtained when an access panel is opened, a sensorreading must be taken only when a device is not turned on, etc.), and soforth.

Regardless of the particulars of (and/or terminology used to describe)cue features, cues, steps, scripts, etc., considerable flexibility maybe permissible in many instances. For example, it may not be required toexecute all steps and/or cues sequentially or continuously in time, atthe same position in space, by the same operator, etc. To refer again toFIG. 2A the step 212 for addressing the procedure could be performedwell away from the machine that is to be inspected; the step 212 asshown advises the operator that inspection is to be conducted, but theoperator may not be (and may not be required to be) near the machine atthe time step 212 is executed. Other procedures may be executed (inwhole or in part) or other actions taken by the operator beforeproceeding to step 214. Similarly, referring to FIG. 2B the step 220 forexamining the component may take place some distance from the rest ofthe machine, i.e. at a location more suitable for measuring the wearstrip as referenced by cue 220A.

Thus while a given script, step, cue, and/or feature thereof may specifyor at least consider factors such as location, time, a particularoperator, etc., uniformity and/or continuity of location, time,operator, etc. is not necessarily required, and indeed non-uniformity ornon-continuity (e.g. executing different steps at different places, atdifferent times, with different operators, etc.) may be useful anddeliberate.

At this point, it may be useful to consider what form scripts, steps,cues, features, etc. may take. As noted, while a script may beillustrated or represented as a flowchart, the providing procedureguidance and confirming procedure execution may resemble a flow chart interms of the actual structure thereof, to the point that it may beaccurate to refer to a script as an “active flow chart”. Thus it may beilluminating to extend the illustrative notion of a flow chart so as toconfigure a script in the form of “modules” of executable instructions.However, it is emphasized that such an approach is an example only, andis not limiting.

With regard to an “active flow chart” implementation, a generalizedscript framework could be established that includes executableinstructions instantiated on the processor for handling aspects ofexecuting the script that may apply to many different scripts and/orsteps therefor, such as protocols for delivering output (e.g. from anoutput cue) to a display screen (or to a device manager for that displayscreen, to an operating system also instantiated on the processor,etc.). Thus, general inputs/output capabilities, processingcapabilities, data logging, hardware management, etc. may be part of ascript framework and/or handled through a script framework, rather thannecessarily being particular to a given script (or steps thereof, etc.).

Continuing with the example, basic elements for scripts may beestablished in the form of “modules” of executable instructions. Ratherthan constructing each script step from scratch, a template ofexecutable instructions may be created, with provision to enable linksto be engaged therewith, cues to be associated therewith, and so forth.Such a template (or “class”) may be instanced so as to produce modulesof code representing script steps. Such an approach may sometimes bereferred to as “syntactical sugar”, wherein generalized classes arecreated for steps, and scripts then may be created by reproducing thoseclasses (e.g. copying the executable instructions and/or data of theclasses) to yield steps that may form the larger script. Creating andconfiguring a step then may be a matter of specifying what cues may bepart of (or associated with) that particular step, which links engagewith the step, and so on.

Similarly, links, cues, cue features, and so forth also may beconfigured as modules of executable instructions. That is, a script maybe a discrete (or at least semi-discrete) assembly of executableinstructions that provide the overall functionality of each step otherdiscrete assemblies of executable instructions that provide the overallfunctionality of links, yet other discrete assemblies of executableinstructions that provide the functionality of various cues, and stillother discrete assemblies of executable instructions that provide thefunctionality of assorted cue features, with the whole disposed in aframework of executable instructions that enables the various discreteunits to cooperate to function as a script. The whole may be referred toas “modular”, in that an instance of a step module may be engaged withinstances of link modules, instances of cue modules may be associatedwith that step module, instances of cue feature modules may beassociated with the various cue modules, etc.

Such an arrangement also may be viewed as a kit of parts adapted to beassembled together, wherein various modules may be selected and engagedtogether to form a functioning script. For example, steps may representinstances of available classes (or templates) of step modules, links mayrepresent instances of available classes of link modules engaged withthose steps, cues may represent instances of available classes of cuemodules associated with those steps, cue features may representinstances of available classes of cue feature modules associated withthose cues, etc. However, full modularity is not required; cues maysimply be selection options within steps, cue features may be selectionoptions within cues, etc.

As may be understood, such an arrangement may at least conceptuallyresemble a flowchart such as shown in FIG. 2A and FIG. 2B, wherein stepsare represented by similar rectangular blocks (“instances” of arectangular shape as may represent a “step class”), cues also arerepresented by similar folded corner rectangular blocks (“instances” ofa folded corner rectangular shape as may represent various “cueclasses”), etc.

Thus, although the arrangement in FIG. 2A and FIG. 2B is illustrative,and the script 200 depicted therein would not necessarily be a flowchart in actual fact, an embodiment of a script similar to the script200 shown may indeed be structured at least somewhat similar to aflowchart. Each step shown may be a broadly similar module of executableinstructions, each link may be a similar module of executableinstructions, each cue may be a module of executable instructions (withoutput cues being similar to one another, input cues being similar toone another—as visually distinguished in FIG. 2A and FIG. 2B by thelocation of the corner folds—etc.).

FIG. 3 shows a script 300 is shown in flowchart form for the arrangingof a script for guiding a procedure and confirming execution, accordingto an embodiment. Now with reference to FIG. 3 , as noted previously(e.g. with regard to FIG. 2A and FIG. 2B) scripts may be simple and/orlinear, but such is not required, and scripts may be complex and/ornon-linear. FIG. 3 shows another example of a script, more complex thanthat in FIG. 2A and FIG. 2B and also non-linear in generalconfiguration. For simplicity, FIG. 3 refers conceptually to steps, etc.rather than showing concrete actions for those steps, so as to plainlyillustrate examples of certain aspects of script configuration. Theconcepts illustrated may apply to a wide variety of different scriptsand configurations, regardless of the particulars of the steps. As notedpreviously scripts may or may not, in fact, be configured as flowcharts, but flowcharts may prove illustrative for the example of FIG. 3.

FIG. 3 shows two types of steps for the script 300 therein, action stepsand decision steps. (As well as a “complete” state 346, which in someinstances optionally may be classified as a step.) The distinctionbetween action steps and decision steps (and likewise any other types ofsteps) is to at least some degree arbitrary and is made for illustrativepurposes.

For the purposes of FIG. 3 , a script step engaged with more than onelink leading therefrom—reflecting more than one possible outcome of anevent in a procedure as reflected by that script step—may be consideredas a decision step. That is, if two or more outcomes for a given stepare possible (e.g. two or more available subsequent steps, steps thatmay or may not be executed, etc.), then that step may be considered adecision step. It is not necessarily implied that a human is making adecision (though human decisions are not excluded); rather, a “decision”may represent a processor selection (e.g. based on executableinstructions), may represent two or more possible states for a procedure(e.g. a part is broken or that part is not broken), etc.

Conversely, for purposes of FIG. 3 a step is identified as an actionstep if the step has only one link leading therefrom, thus the step isexecuted (or skipped, etc.) but no alternatives are presented and thusno decision is implied.

However, it should not be presumed that “nothing happens” in decisionsteps merely because decision steps are not action steps, or that actionsteps necessarily lack any internal options or selections by virtue ofnot being decision steps. As noted, the distinction reflects the overallconfiguration of the script 300, and should not be understood asotherwise limiting.

In the script 300, an action step 302 (referred to generically, withinparentheses) is shown. The action step 302 may represent substantiallyany action including but not limited to steps similar to those alreadydescribed (e.g. with regard to FIG. 2A and FIG. 2B), for example openinga device, removing a part, taking measurements, etc.

When the action step is complete, the script 300 follows a link 301 fromaction step 302 to another action step 304. Likewise, upon execution ofaction step 304, the script follows a link 303 engaging action step 304to a decision step 306.

The decision step 306 has two links 305 and 315 engaged therewith andleading to subsequent steps (namely action steps 308 and 320,respectively). Thus, some selection—whether an operator decision, aprocessor determination, a selection based on the status of theprocedure itself or the state of the environment, etc.—is made, with twopossible outcomes: decision A is reached and link 305 is followed toaction step 308, or decision B is reached and link 315 is followed toaction step 320.

For instances wherein decision A is reached, link 305 leads to actionstep 308. Action step 308 is then executed, and link 307 is followedtherefrom.

As may be seen, link 307 in FIG. 3 engages not with an individual stepbut with a step group 310. As noted previously, steps may be configuredas modules. As may be seen from FIG. 3 , arrangements of steps, links,etc. also may be grouped together. Within the step group 310, actionsteps 312, 314, 316, and 318 execute in sequence following links 309,311, and 313 as previously described with regard to other action stepsin FIG. 3 . With step group 310 complete, the script then proceeds alonglink 323 as shown.

Thus, a step group may be understood to execute similarly to individualsteps. In practice such grouping may or may not be visible to anoperator executing the script 300, however, in terms of the overallconfiguration of the script 300, it may be useful to treat assemblies ofsteps, links, etc. as larger units or groups.

Such an arrangement may be useful for example in creating new scripts,and/or modifying existing scripts. For example, procedures formanufacturing two closely related models of an automobile may besimilar, at least in part. If both vehicles have identical dashboards,those portions of an assembly procedure relating to dashboard assemblylikewise may be identical; consequently, for a script corresponding toassembly procedure the portions thereof that correspond with dashboardassembly also may be identical.

Even if a given portion of two different procedures is not identicalbetween those procedures, those portions of the procedures neverthelessmay be represented by identical sequences within a script; if a hubcapis held on with six bolts rather than five, and the script does notspecify the number of bolts, the scripts may be identical even thoughthe procedures are not.

Furthermore, if a group of steps would appear repeatedly in a script, itmay be useful to configure that group of steps once, and then “call” thegroup each time. From the point of view of the operator, the script mayappear the same either way, but the script itself may be simpler and/ormore compact.

In addition, even if portions of a new script differ from portions of anold script (e.g. because a new procedure differs from an old procedure),it may be expedient to copy a portion of script—e.g. a group of steps as310—from one script to another, as a cohesive group rather than copyingor reproducing each individual step. That group then may be modified tocorrespond so that the new script corresponds to the new procedure,without the need to recreate the entire group.

Regardless, the step group 310 as shown in FIG. 3 is illustrative, andnot limiting. Other arrangements of and/or applications for groups andgrouping also may be suitable.

Moving on in FIG. 3 , If decision B is reached in decision step 306 thenlink 315 engaged therewith is followed leading to subsequent action step320. The script then proceeds with link 317, action step 322, link 319,action step 324, and link 321.

Regardless of whether decision A or decision B is reached in decisionstep 306, eventually the script arrives at action step 326 either vialink 321 (from step group 310) or link 323 (from action step 324). It isnoted that other arrangements may be suitable; it is not required forscripts to “come back together”, and different decisions may result infollowing portions of a script that never rejoin, that terminateseparately, that link to other scripts, etc.

With action step 326 executed, the script proceeds along link 325engaged therewith to reach decision step 328. As with decision step 306,some determination is made in decision step 328, reaching eitherdecision C or decision D. (Although the arrangement of FIG. 3 showsdecision steps with exactly two outcomes, this is not limiting, andother arrangements may be suitable.)

If decision C is reached, the script follows link 327 from decision step328 to action step 330. Upon executing step 330, link 329 is followed(leading to action step 342).

However, if decision D is reached, the script follows link 331 fromdecision step 328 to action step 332. Action steps 332, 334, 336, and338 are executed in sequence following links 333, 335, and 337therebetween.

Once action step 338 is complete, link 339 leads to another decisionstep 340. A determination is made in decision step 340 reaching eitherdecision E or decision F. As may be seen, if decision F is reached thescript then loops back to action step 334 along link 341. Thus as shown,loops (as well as other non-linear architecture) may be present invarious script embodiments. Indeed, the scripts are not limited in termsof configuration or complexity and may include substantially any numberand/or type of conceivable arrangements for steps, links, etc. So longas a given script functions and at least substantially corresponds tothe appropriate procedure, scripts may be arbitrarily complex.

If decision E is reached in decision step 340 (or if decision C werereached previously at decision step 328), the script proceeds to actionstep 342 via link 343 (or likewise to action step 342 from action step330 via link 329). Once action step 342 is executed, the script iscomplete; although a link 345 to a completion indicator 346 is shown forclarity in FIG. 3 , it may be equally suitable simply to end the scriptat action step 342 (thus excluding link 345 and completion indicator346).

It is emphasized that the method in FIG. 3 is an example only; otherarrangements, whether simpler, more complex, or of similar complexity,also may be suitable.

Now with reference to FIG. 4A and FIG. 4B, as noted various embodimentsand arrangements of scripts may be executed, in whole or in part byoperators such as persons wearing a head-mounted display or using someother portable electronic devices, such that executing the scripts atleast substantially correspond to executing various procedures. Suchscripts also may be created, modified, etc.; in such instances, personscreating and/or modifying the scripts may be referred to as “authors” ofthose scripts. FIG. 4A shows a flowchart 400 for the arranging of ascript for guiding a procedure and confirming execution with regard to ascript author, according to an embodiment. FIG. 4B shows a continuationof the flowchart 400 in FIG. 4A for the arranging of a script forguiding a procedure and confirming execution with regard to a scriptauthor, according to an embodiment. As noted previously with regard tothe execution of scripts, it may be useful to provide brief explanationsof certain terms associated with the creation and modification ofscripts.

An “author” refers to a person creating or modifying a script. Forexample, an aircraft mechanic creating a script to correspond with therepair or maintenance of an aircraft may be considered the author (or atleast an author) of that script. Similarly, another mechanic who laterupdates that script (e.g. to accommodate changes in hardware, new workstandards, etc.) also may be considered an author of that script.Authors may vary greatly and are not limited. As with an operatorexecuting a script, it is not required for an author to create or modifyan entire script for a given procedure; two or more authors maycooperate to produce or change a script (potentially with those authorsdoing so at different times, in different places, through differentdevices, using different interfaces, etc.).

A “step avatar” refers to some representation of a step in a script, andassociated with that step, such that placing the avatar corresponds withplacing the step. For example, a flowchart icon may serve to represent astep; such an icon might be placed and/or manipulated spatially within a“desktop” style graphical interface so that authoring the script atleast somewhat resembles configuring a flow chart. However, step avatarsare not limited, and other arrangements may be suitable. For example, amenu, control panel, etc. as may be displayed as an augmented realityobject (e.g. on a head-mounted display) also may serve as an avatar; insuch case the step avatar may be placed within a script while actuallyperforming the procedure in question, such as by calling up anappropriate menu during each event of the procedure so as to create astep in the script that corresponds with that event. (Such anarrangement may be useful in creating scripts that match the procedureas actually performed, as opposed to configuring a flow chart based onan “on paper” abstraction of the procedure.) In other embodiments, atext command or a block of code in a command line interface thatspecifies or represents a step may be considered to be an avatar forthat step.

A “step avatar class” refers to a format for representing a step avatar.For example, considering again a flowchart icon as a step avatar, a stepavatar class for that step avatar may be executable instructions thatdefine the shape of the avatar, the behavior of the avatar within aninterface (e.g. enabling drag-and-drop movement), etc. In such case,each step avatar may be considered as an instance of the step avatarclass. In colloquial terms, the step avatar class may be considered a“master” or “original”, with the individual step avatars being “copies”(or instances) thereof.

Steps have already been described herein.

A “step class” refers to a format for a step, with an arrangementbetween steps and step classes at least somewhat similar to thatdescribed for step avatars and step avatar classes. That is, the stepclass may define how steps actually execute, how cues are accommodatedwithin (or otherwise engaged with) a step, and so forth. The step classthus in some sense defines the behavior and capabilities of the steps ina script.

Similarly, a “link avatar”, an “output cue avatar”, and an “input cueavatar” refer to representations of links, output cues, and input cues,respectively. Likewise, a “link avatar class”, an “output cue avatarclass”, and an “input cue avatar class” refer to formats, definitions,etc. for link avatars, output cue avatars, and input cue avatars,respectively. Similarly, a “link class”, an “output cue class”, and an“input cue class” refer to formats, definitions, etc. for linksthemselves, output cues themselves, and input cues themselves,respectively. (Links, output cues, and input cues have already beendescribed herein.)

Collectively, the various classes and instances as described above maybe considered to be a set of tools or components for creating a script.As a colloquial example, if one were to consider physical bricks asinstances of steps, then building a wall out of those bricks (“linked”in a particular arrangement with mortar, etc.) may be comparable toauthoring a script from steps.

For simplicity, certain examples herein refer to scripts as having asingle step class, a single output cue avatar class, and so forth.However, embodiments are not limited only to single classes for eachelement. For example, different types of steps (action steps, decisionsteps, etc.) may have different step classes, and such different stepclasses may be represented by different step avatar classes. Consideringagain a desktop graphical interface as an example, this may representseveral types of the step being available for use within a script, witheach type of step being represented by a different icon (e.g. adifferent shape, color, etc.).

With regard to interface, it should be understood that to at least somedegree the nature of the various classes and instances thereof maydefine the interface in which those classes and instances aremanipulated. Conversely, it may also be said that the classes andinstances may to some degree define the interface. (In some sense,creating fully functional avatar classes may be considered to in fact bethe creation of a functional interface.) Thus, a desktop graphicalinterface may be associated with desktop-style graphical icon avatars, agraphical augmented reality interface may be associated with graphicalaugmented reality menus, a command line interface may be associated withcommand line avatars, etc. However, the particulars of an interface arenot otherwise limited.

Referring again to FIG. 4A, tools for authoring 400 a script aredisposed 402 onto the processor. A number of such tools may be disposedonto the processor in such fashion. It is noted that the order ofdisposing such tools and/or the manner by which those tools are divided(e.g. step avatar classes being described as distinct from step classes)in the example of FIG. 4A is illustrative, and should not be understoodas limiting.

Within step 402 and as a sub-step thereof, a step avatar class isdisposed 402A on the processor. A step class may, for example, includeexecutable instructions and/or data, be loaded to the processor.

It is noted that a distinction is made in terminology between“instantiating instances” on a processor and “disposing classes” on theprocessor. Although in a strictly technical sense instantiating aninstance of a class on a processor and loading the class itself onto theprocessor may involve similar or identical loading of data andexecutable instructions, for descriptive purposes the terms“instantiating” and “loading” are treated separately, for clarity. Thus,a class is disposed; an instance of that class is instantiated. Thedistinction may be noteworthy at least in terms of the method itself;for example, while disposing various classes onto a processor may happenautomatically or entirely within a processor (e.g. a block of softwareself-installs, uploads from a hard drive, downloads from a cloud, etc.),instantiating instances of those classes typically is done at least inpart by a human author, that author creating various instances ofvarious avatars so as to configure steps, links, cues, etc. associatedwith those avatars into a coherent and functional script.

Moving on in FIG. 4A, a step class is also disposed 402B on theprocessor. Step avatar classes and step classes have been describedpreviously. The step class is then associated 402C with the step avatarclass on the processor, such that instantiating a step avatar from thestep avatar class will also instantiate a step from the step class onthe processor therewith. That is, when an author creates a step avatar,an appropriate step itself is created; when the author moves,rearranges, modifies, etc. that step avatar, that step itself is moved,rearranged, modified, etc.

Similarly, a link avatar class is disposed 402D on the processor, a linkclass is disposed 402E on the processor, and the link class isassociated 402F with the link avatar class such that instantiating alink avatar from the link avatar class will also instantiate a link fromthe link class therewith.

An output cue avatar class is disposed 402G on the processor, an outputcue class is disposed 402H on the processor, and the output cue class isassociated 402I with the output cue avatar class such that instantiatingan output cue avatar from the output cue avatar class will alsoinstantiate an output cue from the output cue class therewith.

An input cue avatar class is disposed 402J on the processor, an inputcue class is disposed 402K on the processor, and the input cue class isassociated 402L with the input cue avatar class such that instantiatingan input cue avatar from the input cue avatar class will alsoinstantiate an input cue from the input cue class therewith.

As noted previously, multiple types of steps and links, additional typesof cues, etc. may be suitable for at least certain scripts. Likewise,for authoring such scripts, multiple step classes and step avatarclasses may be disposed and associated on the processor, additional cueclasses and cue avatar classes may be disposed and associated on theprocessor, etc. Embodiments are not limited only to the arrangementshown in FIG. 4A.

Moving on to FIG. 4B, the author arranges 404 instances of the variousclasses on the processor. As noted, each script may include steps, linksamong those steps, output cues, input cues, etc. Thus, as shown in FIG.4B arranging 404 instances to author a script may include instantiating404A one or more step avatars onto the processor, in whatever interfacemay be suited for the particular avatars of a given embodiment, so as toinstantiate steps of the script associated therewith (those steps, inturn, corresponding to events in a procedure). One or more link avatarsare instantiated 404B onto the processor in the interface, so as toinstantiate links for the script associated therewith. One or moreoutput cue avatars are instantiated 404C onto the processor in theinterface, so as to instantiate output cues for the script associatedtherewith (with those output cues also being associated with a givenstep, for example). One or more input cue avatars are instantiated 404Donto the processor in the interface, so as to instantiate input cues forthe script associated therewith.

Thus, in instantiating step avatars, link avatars, output cue avatars,and input cue avatars (e.g. creating, manipulating, moving,interconnecting, etc. the avatars themselves), the author is in turnarranging steps, links, output cues, and input cues so as to form ascript such that executing the script at least substantially correspondsto executing the procedure.

For illustrative purposes, it may be useful to describe as an examplewhat the author may literally see, and/or literally do when authoring ascript. As noted, avatars may take many forms, and manipulations ofavatars may vary considerably from one embodiment to another. However,in at least certain instances a “flow chart” paradigm may be useful.Thus, an author may view and interact with an arrangement that isconceptually and/or visually similar to that in FIG. 3 : action stepsmay be represented by rectangular action step avatars, decision steps bydiamond shaped decision avatars, links by link avatars in the form ofarrows, etc. Cues may or may not be individually visible; in certainembodiments, cues could be dragged and dropped onto individual actionsteps, and/or action steps might be “opened” or enlarged (e.g. bydouble-clicking the avatar for a given action step), etc. Steps, links,and the like may be dragged around the screen, pasted, cut, copied, etc.as in a desktop graphical interface. (Indeed, such actions could becarried out in just such an interface on a desktop PC, although otherarrangements also may be similar.)

FIG. 5A shows an interface for authoring a script, according to anembodiment. FIG. 5B shows an interface for executing a script, accordingto an embodiment. FIG. 5A may resemble what an author might see whenauthoring a script and FIG. 5B may resemble what an operator might seewhen executing a script. However, it is emphasized that the arrangementsin FIG. 5A and FIG. 5B are examples only and are not limiting.

As may be seen in FIG. 5A, a script is shown therein, in the form of aflowchart (though the form of a flowchart is itself an example only, andis not limiting). The script exhibits seven steps 502, 506, 510, 514,518, 522, and 526, engaged with and/or sequenced by six links 504, 508,512, 516, 520, and 524. As may be seen the script in FIG. 5A correspondsto an automobile-related procedure. Broadly speaking the script proceedsfrom step 502 marked “Open hood” to step 506 marked “Change oil” vialink 504, and then by link 508 to step 510 marked “Will the car start?”As may be understood, step 510 represents a decision to be made (andthus step 510 may be considered a decision step): the script branchesalong alternate links 512 and 520. If the car does not start (as shownby the “NO”), the script proceeds along link 512 to step 514, markedwith “0123456789”, and then along link 516 to step 518, marked with “Tryagain”. However, if the car does start (as shown by “YES”), the scriptproceeds along link 520 to step 522, marked with “Vehicle release viasystem web app”, then along link 524 to step 526, marked with “Driveaway”.

In addition, as may be seen several graphical elements are present inFIG. 5A that are not necessarily steps or links in a strict sense, butthat may still serve purposes related to the script. A circular starticon element 530 marked with “Start” and another circular icon 534marked with “End” may be seen. Similarly, arrows 532, 536, and 538 arevisible, engaging the start and stop icons elements 530 and 534 withsteps in the script.

Depending on the particulars of a script, an interface, etc., suchelements as 530 through 538 may or may not be considered steps and/orlinks per se. A “start step” may have no actions, inputs, outputs, etc.associated therewith, and may merely be a convenience (whether for theauthor in scripting, the architect in creating script tools, etc.);likewise an “end step” may have no actions, inputs, outputs, etc..Similarly, if elements 530 and 534 are not considered steps, then arrows532, 536, and 538 may not be considered links in that the arrows do notconnect one step to another (assuming a given embodiment defines linksas only linking active steps, which may not be the case for allembodiments). However, embodiments may vary in this regard; it is notrequired that “start” or “end” steps be included, nor that a start orend feature be considered a step, nor otherwise, that only steps andlinks be present in a script, etc. Scripts, interfaces, and toolstherefor may vary considerably and are not limited.

Several example features also are noted in FIG. 5A.

As noted previously, different types of steps may be distinguished fromone another, such as by shape, color, etc.. Such an arrangement may beconvenient to an author (e.g. as shown, the decision step 510 is darkerand differently shaped than other non-decision steps), and/or may serveother purposes. Such distinctions (or other distinctions), while notrequired, potentially may be useful for at least certain embodiments.

In addition, it is noted that certain steps therein show icons in thebottom right corner thereof. For example, 502 and 522 show chain links,506, 516, and 526 show paperclips, and 514 shows a bar code. Such icons(or other indicators) may serve to graphically represent features of orrelating to the steps. Thus, chain links may indicate a link tosomething outside the script with respect to steps 502 and/or 522, suchas linking to a separate script for opening the hood in 502 (e.g.depending on the vehicle type), activating the system web app referencedin 522, etc. Similarly, paper clips may represent files attached to thestep and/or files that are to be attached to the step, such as aphotograph verifying that the oil is being changed in 506. Likewise, thebar code may indicate recording numerical and/or coded information (suchas the VIN or license plate number for the vehicle that won't start, forfuture reference), etc. Depending on usage and interface configuration,such icons may represent inputs to be acquired, outputs to be delivered,both, neither, etc. These are examples only, and other arrangements maybe suitable.

Similarly, the text shown in steps 502, 506, 510, 514, 518, 522, and 526may represent output to be delivered to an operator e.g. via an outputcue, but alternatively or in addition may represent some otherinformation, such as being a “title” or “flag” for assisting authors inrecognizing the steps. As a more concrete example, “Change oil” in 506may be actual text that is presented to an operator, or may be a“handle” for the script's author to refer to a step that includes manydifferent input cues, output cues, etc. therein (although individualcues may not be individually shown in such a “high level” view of step506), or may serve some other function.

In addition to steps 502, 506, 510, 514, 518, 522, and 526, links, 504,508, 512, 516, 520, and 524, etc. as shown in FIG. 5A, author interfacesmay include—as is visible in FIG. 5A—various other elements, such aswindows, icons, menus, logos, etc. Interfaces and/or authoring therewithare not limited only to the use of steps and links, nor to anyparticular configuration or tools for steps and links.

Turning to FIG. 5B, therein is shown an illustration of an exampleinterface for authoring executing a script. As may be seen, two lines oftext 550 and 552 are visible in FIG. 5B, near the top of the gray-tonedregion on the left. In the arrangement shown, it may be considered thattext 550 represents a step title, or a procedure title, etc., while text552 may represent a text output (e.g. as delivered via an output cue).However, depending on precisely how steps, output cues, output, etc. areconsidered for a given embodiment, it may be reasonable to consider text550 to be output from an output cue as well, etc.

In the arrangement of FIG. 5B, the overall arrangement is one a displayfield (e.g. for text) on the right, an open space in the center, a rowof menu icons (e.g. for navigation, image and/or file attachment, etc.)below the display field, and a column of menu icons on the right. Suchan arrangement may be suitable for example for an augmented realitysystem such as a head-mounted display, in that the operator's centralvision may remain clear to allow viewing of the work being done (giventhe text 550 and 552, possibly some form of pump maintenance), whileproviding controls accessed e.g. via gesture input, and display spacefor instructions such as text 550 and 552 and/or other information.However, as with the author interface in FIG. 5A, the operator interfacein FIG. 5B is an example only and is not limiting.

FIG. 6 shows a flow chart 600 for providing guidance during a procedureand for confirming the execution of that procedure with regard to ascripting architect, according to an embodiment. Where certain previousexamples have shown the execution of a script by an operator, andcreation of a script by an author, the arrangement in FIG. 6 refers tothe provision of tools for an author to produce or modify a script thatthen may be executed by an operator. For purposes of description, such aperson providing script authoring tools may be referred to as anarchitect. Thus, broadly speaking, an architect provides the“infrastructure” for authoring a script; an author creates the scriptitself; and an operator executes the script, such that executing thescript at least substantially corresponds with executing a procedure.

In the arrangement of FIG. 6 , a step avatar class is defined 602 on aprocessor. As noted previously, a “step avatar class” refers to a formatfor representing a step avatar (e.g. a graphical appearance to indicatea step), such as may be used for conveniently adding, deleting, andmodifying steps while authoring a script.

Typically though not necessarily, the step avatar class is defined 602by an architect, such as a human computer programmer. However, it shouldbe understood that the nature of the architect, and the manner by whichan architect may define 602 a step avatar class (and likewise carry outother actions in FIG. 6 ), may vary considerably from one embodiment toanother. For example, an architect may be a single individual, may beseveral individuals cooperating together as a group, may be a series ofindividuals and/or groups working in different times and/or places (e.g.defining 602 different revisions of the step avatar class over time, inresponse to changes in processors, operating systems, user requirements,etc.). Likewise, defining 602 an avatar class may include creatingexecutable instructions “from scratch”, editing together pre-writtencommands, subroutines, etc., integrating functional “blocks” ofinstructions or “packages” of code (possibly in a pre-existing codinginterface), etc. In principle defining 602 an avatar class could becarried out at a fundamental machine-code level, even to the point ofcoding 0s and 1s; in practice, however, it may also be suitable todefine 602 an avatar class within some existing framework, such astaking advantage of an existing operating system, using and modifyingcommands and executable instructions therein, adding such commands andinstructions as necessary, etc. As a more concrete example, an avatarclass may be defined 602 within Android or some other mobile operatingsystem, or within Windows or some other PC operating system, using theinfrastructure and conventions thereof.

Continuing in FIG. 6 , a step class is defined 604 on the processor. Asnoted previously, a step class refers to a format for a step in ascript, that step at least substantially corresponding to an event in aprocedure.

The step class is associated 606 with the step avatar class in theprocessor, such that disposing/manipulating a step avatar within aninterface may dispose/manipulate a step within a script (as previouslydescribed). It is noted that in certain previous examples (such as FIG.4 ), a step avatar class may be associated with a step class inauthoring the script. Thus an author may so associate classes withrespective avatar classes, therefore, as part of authoring the script.Such an arrangement may facilitate customization of avatars, forexample; that is, an author may choose to utilize a rectangular avatarfor certain steps or groups of steps, a diamond-shaped avatar for othersteps or groups of steps, different colors of avatars, etc., based onpersonal preferences and/or other factors.

However, as shown in FIG. 6 it may be equally suitable to associate 606a step class with a step avatar class as part of creating the tools forauthoring scripts, rather than as part of authoring the script itself.Thus, some or all such associations may be implemented by an architect,rather than an author. However, such association may be “mixed andmatched”, i.e. some classes may be associated with avatar classestherefor by an architect while others are associated by the author, anarchitect may perform such association while an author may change theassociation, etc.

Moving on in FIG. 6 , a link avatar class is defined 608 in theprocessor, for example by an architect. A link class is also defined 610on the processor, and the link class is associated 612 with the linkavatar class. As noted, a link avatar class refers to a representation(e.g. a graphical image) for various links as may exist within a script.

An output cue avatar class is defined 614 on the processor, an outputcue class is defined 616 on the processor, and the output cue class isassociated 618 with the output cue avatar class. As noted, an output cueclass refers to a configuration for output cues, and an output cueavatar class refers to a representation for output cues.

Similarly, an input cue avatar class is defined 620 on the processor, aninput cue class is defined 622 on the processor, and the input cue classis associated 624 with the input cue avatar class. Again as noted, aninput cue class refers to a configuration for input cues, and an inputcue avatar class refers to a representation for input cues.

Still with reference to FIG. 6 , an interface is defined 626 in theprocessor. In certain previous examples (such as FIG. 4 ), an interfacemay be assumed as presently given the existence and manipulation ofvarious classes and instances, may be considered to be part of orinherent to the definition of various classes, instances, etc. (forexample, if classes exist, the classes enable formation of instancestherefrom, manipulation of those instances is supported to author ascript, etc., then enabling those functions may in itself constituteproviding an interface), and so forth.

However, as shown in the arrangement of FIG. 6 , the definition 626 ofan interface may also be explicit. That is, some “space” may be defined(such as a window, a desktop, an augmented reality region, etc.), rulesmay be defined for that space with regard to how entities therein behaveand are treated, and so forth. In such instance, some of thefunctionality associated with various classes and instances may,strictly speaking, be part of the interface itself. That is, theappearance (e.g. the size, color, etc.) of an instance of a step avatarclass may be defined at least in part by the particulars of theinterface, the behaviors and even association with an instance of a stepclass may be enabled by the interface, etc.

As with the various classes and instances, the interface itself may bedefined 626 in a variety of manners. For example, an interface may becoded “from scratch”, or may be adapted or even used as-is if a suitableinterface already exists. As a more concrete example, an operatingsystem such as Windows or Android may support the manipulation ofvarious avatars in the form of graphical icons, and such functions maybe utilized either as an interface or as some portion thereof. Otherarrangements also may be suitable, and the nature of the interface andmanner in which it is provided or defined is not limited.

FIG. 7 shows an apparatus 700 for providing guidance for or during aprocedure and for confirming the execution of that procedure with regardto a scripting operator, according to an embodiment. The arrangementshown in FIG. 7 refers to script execution, as previously describedherein; thus, the script is being followed so that a procedure iscarried out (e.g. at least in part by an operator).

As may be seen, the example apparatus 700 includes a processor 702. Asnoted previously, suitable processors may vary considerably, and mayinclude but are not limited to digital processors such as may be foundin smartphones, head-mounted displays, and other portable electronicdevices, and as may be found in desktop and laptop computers.

An outputter 704 is disposed in communication with the processor 702.The outputter 704 also may vary considerably. The outputter 704 isadapted to deliver output cues (and potentially other content) to anoperator (not shown); the outputter 704 also may be adapted to delivercontent to targets other than an operator. As noted, output cues mayvary considerably, including but not limited to displayed text, images,graphics, animations, video, etc. as well as audio and other formats.Consequently, an outputter 704 may vary considerably as well, dependingon the particulars of what output cues are to be delivered and what formthose output cues may take. For example, suitable outputters 704 mayinclude but are not limited to visual displays such as LED, OLED, LCD,CRT, etc., whether stereo or mono, color, grayscale, or monochrome, etc.and as incorporated into a variety of forms such as PC monitors,hand-held electronic device displays, head-mounted near-eye displays,and so forth. Other arrangements also may be suitable. In FIG. 7 , theoutputter 704 is shown as a graphical mono display such as a CRT, butthis is an example only; it is also noted that a given apparatus 700 isnot limited to only a single outputter 704 as in the example shown.

An inputter 706 also is disposed in communication with the processor702. The outputter 706 likewise may vary considerably. The inputter 706is adapted to accept input cues (and potentially other content) from anoperator; the inputter 706 also may be adapted to accept content fromsources other than an operator. As also noted, input cues may varyconsiderably, including but not limited to gesture inputs, voice inputs,keyboard or other text inputs, image and/or video captures, etc. Thus aninputter 706 likewise may vary considerably, depending on the intendedform, source, and nature of the input cues. For example, suitableinputters may include but are not limited to digital imagers such as RGBcameras and depth cameras, audio microphones, keypads, mice, etc. Suchinputters 706 may take a variety of forms and/or be incorporated into arange of other systems, such as head mounted displays, smartphones, etc.Although the inputter 706 in FIG. 7 is shown as an imager such as adigital time-of-flight depth camera, this is an example only. Also, agiven apparatus 700 is not limited only to a single inputter 706 asshown. Furthermore, it is noted that to at least some extent theinputter 706 (and likewise an outputter 704 above) may be or includeelements other than hardware. For example, if hand gestures areanticipated as inputs, an inputter 706 may include a physical sensoradapted to receive images or other data of an operator's hands; howeverother components (such as executable instructions disposed on theprocessor 702) may be present for detecting and recognizing gestureinputs within such images. Although the inputter 706 is shown as asingular, discrete element, in practice the inputter 706 and/or otherelements of the apparatus 700 may take multiple forms, be distributed orsubdivided into subcomponents, etc. Other arrangements also may besuitable.

Still, with reference to FIG. 7 , a script 708 is instantiated on theprocessor 702. Typically though not necessarily, the script 708 may be adata entity, formed of executable instructions and/or data, etc.Regardless of form, the script 708 is adapted such that executing thescript 708 substantially corresponds to executing a procedure.

The script 708, in turn, includes at least one step 710, link 712,output cue 714, and input cue 716. (Only one each is shown forsimplicity, though many such may be and typically will be present.) Thesteps 710, links 712, output cues 714, and input cues 716 likewise maybe data entities, formed of executable instructions and/or data, etc. Asdescribed previously, the steps 710 correspond with events of theprocedure, the links 712 correspond with the sequencing of the events,the output cues 714 specify outputs from the processor 702 e.g. forproviding procedure guidance to an operator, and the input cues 716specify inputs to the processor 702 e.g. for confirming procedureexecution.

In the apparatus 700, the processor 702 is adapted to execute the script708 beginning with an initial one of the steps 710, executing thatinitial step 710 (with at least some steps being at least partlyoperator-implemented), and following links 712 to other steps 710 as thevarious output and input cues 714 and 716 are satisfied, until executionof the script 708 has been completed. Thus, in cooperating to executethe script 708, the processor 702 and an operator cooperate to executethe corresponding procedure.

FIG. 8 shows an apparatus 800 for providing guidance for or during aprocedure and for confirming the execution of that procedure with regardto a scripting author, according to an embodiment. The arrangement shownin FIG. 8 refers to script creation and modification, as previouslydescribed herein; thus, the script is being configured (e.g. at least inpart by an author) so as to correspond with a procedure, such that whenthe script is followed that procedure is carried out (e.g. at least inpart by an operator).

The example apparatus 800 includes a processor 802, an outputter 804disposed in communication with the processor 802, and an inputter 806also disposed in communication with the processor 802. At least in termsof the processor 802, outputter 804, and inputter 806, the apparatus 800may resemble an apparatus 700 as shown in FIG. 7 , although theapparatus 700 in FIG. 7 is described with regard to executing scriptswhile the apparatus 800 in FIG. 8 is described with regard to authoringscripts. Indeed, it may be suitable for a single hardware device to beadapted both for executing scripts and for authoring scripts (and/or forarchitecting tools for authoring scripts as also previously described);for example, a given head-mounted display (or another device) may beused both for authoring a script for a procedure and to execute thatscript to perform the procedure (possibly by different individuals atdifferent times and places, etc.). Thus, comments with regard tohardware elements in FIG. 7 may apply similarly to elements in FIG. 8 .In embodiments with such dual functions, an apparatus may includeelements both as shown in FIG. 7 and as shown in FIG. 8 , and/or otherelements; in more colloquial terms, software for running scripts andwriting them may be loaded on the same system (and other software alsomay be present). While such arrangements are not prohibited, neither aresuch arrangements required for all embodiments.

Continuing in FIG. 8 , a number of elements 808 through 824 are showninstantiated on the processor 802. These elements 808 through 824 may bedata entities, e.g. executable instructions and/or data.

In particular, an interface 808 is disposed on the processor. Theinterface 808 is adapted for configuring steps, links, cues, and avatarstherefor so as to form a script, that script being such that executingthe script at least substantially corresponds to executing a procedure.In particular, the interface 808 is adapted to enable an author to soconfigure steps, links, cues, avatars therefor, etc. so as to authorsuch a script.

As shown in FIG. 8 , the interface 808 is a stand-alone element,essentially a “playing field” where the other elements 810 through 824may perform functions. However, it may be equally suitable for theinterface 808 to be considered a “parent” element, that is, for theother elements 810 through 824 to be considered as part of the interface808 itself. Such a distinction may be academic in many if not allembodiments (e.g. being simply different ways of describing functionallyequivalent arrangements), but even if of practical importance thedistinction is not limiting.

A step class element 810 and a step avatar class 812 are disposed on theprocessor 802. The step class element 810 and step avatar class 812 areassociated, such that disposing an instance of the step avatar class 812within the interface 808 also disposes an instance of the step classelement 810 as a step within a procedure. Thus, adding, moving,modifying, or deleting a step avatar within the interface 808 would add,move, modify, or delete an associated step within the script.

A link class 814 and a link avatar class 816 also are disposed on theprocessor 802. The link class 814 and link avatar class 816 areassociated similarly to the step class element 810 and step avatar class812 above, such that disposing an instance of the link avatar class 816within the interface 808 similarly disposes an instance of the linkclass 814 as a link within the procedure.

An output cue class 818 and an output cue avatar class 820 are disposedon the processor 802, as are an input cue class 822 and an input cueavatar class 824. The output cue class 818 and the output cue avatarclass 820 are associated, and the input cue class 822 and the input cueavatar class 824 are associated, such that disposing an instance of theoutput cue avatar class 820 or the input cue avatar class 824 within theinterface 808 also disposes an instance of the output cue class 818 orthe input cue class 822 as an output cue or input cue respectivelywithin the procedure.

In the apparatus 800, the processor 802 is adapted to configureinstances of the step class element 810, link class 814, output cueclass 818, and input cue class 822 in response to configuration (e.g. byan author) of instances of the step avatar class 812, link avatar class816, output cue avatar class 820, and input cue avatar class 824 withinthe interface 808, so as to enable authoring of a script (the script notbeing shown in FIG. 8 ), such that executing that script at leastsubstantially corresponds with executing a procedure.

FIG. 9 shows an apparatus 900 for providing guidance for or during aprocedure and for confirming the execution of that procedure, accordingto an embodiment. As noted with regard to FIG. 7 and FIG. 8 , a givenapparatus may not be limited only to one aspect of providing procedureguidance and confirming procedure execution, e.g. to executing scriptsonly, to authoring scripts only, etc. The arrangement shown in FIG. 9likewise is not necessarily limited only to one such aspect; dependingon particulars, the apparatus 900 may be adapted for facilitating anoperator in executing scripts, facilitating an author in authoringscripts, facilitating an architect in defining tools for authoringscripts, some combination thereof, etc.

As may be seen, the apparatus 900 includes a processor 902. Depending onthe embodiment and which aspect(s) the apparatus 900 may be configuredto carry out, the processor may include data entities disposed thereon,such as one or more scripts with steps, links, output cues, and inputcues (as shown in FIG. 7 ), and/or an interface, step class, step avatarclass, link class, link avatar class, output cue class, output cueavatar class, input class, and input cue avatar class (as shown in FIG.8 ), etc.; however, data entities typically not being visible inperspective view, such data entities are not illustrated in FIG. 9 .

The apparatus 900 also includes the outputters 904A and 904B, configuredas left and right stereo graphical displays in the example embodimentshown. Such the outputters 904A and 904B may for example outputgraphical information to an operator (and/or an author, architect,another target, etc.), such as output cues with text, images,animations, etc. One non-limiting concrete example would be augmentedreality content, displayed overlaying a view of the physical worldthrough see-through displays (e.g. the outputters 904A and 904B).

The apparatus 900 includes inputters 906A and 906B, configured as leftand right stereo imagers in the example shown. Such inputters 906A and906B may, for example, accept visual input information from an operator(or another source), such as input cues with images or video of thephysical world, inputs from an operator, etc. One non-limiting concreteexample would be hand gestures carried out within the field of view ofthe inputters 906A and 906B, with gestures being detected and recognizedwithin captured images (e.g. using executable instructions for gesturerecognition as may be instantiated on the processor 902) as usercommands.

As may be seen, the processor 902, the outputters 904A and 904B, andinputters 906A and 906B are shown engaged with a body 912. The body 912in the example of FIG. 9 is in the form of a pair of glasses, such thatthe apparatus 900 may be worn as a head-mounted display. As may beunderstood, in such configuration with the apparatus 900 is worn theoutputters 904A and 904B may be disposed substantially facing,proximate, and aligned with the wearer's eyes, while the inputters 906Aand 906B may be disposed facing substantially forward with respect tothe wearer's head, e.g. such that the inputters 906A and 906B mayexhibit a field of view at least somewhat similar to that of the wearer(so that the inputters 906A and 906B “see what the user sees”), and/orat least including a region in front of the wearer convenient to thewearer in executing gesture inputs. However, the arrangement shown anddescribed with regard to FIG. 9 is an example only, and otherarrangements also may be suitable.

Embodiments may vary considerably. Although not necessarily required,additional elements are shown for the apparatus 900 in FIG. 9 . Forexample, a communicator 908 is shown disposed on the body 912. Such acommunicator 908 may, for example, enable communication between theapparatus 900 and/or a wearer thereof (e.g. an operator for a script)with other persons, systems, etc. A data store 910 also is showndisposed on the body 912. Such a data store 910 may, for example, enablescripts or portions thereof, classes or instances thereof, etc. to beinstantiated onto the processor 902 from stored memory, may enablerecall information for supporting scripts or creation thereof, may storeinputs as received e.g. from the inputters 906A and 906B, etc.

In addition, depending on the particulars, elements such as thecommunicator 908 and the data store 910 may function as and/or beconsidered as the inputters, the outputters, etc. in their own right.For example, if a machine is being serviced as part of a procedure, andthat machine includes an RFID tag, the RFID tag may be queried by thecommunicator 908 and the information from that RFID tag may be receivedby the communicator; thus the communicator 908 may be considered to haveperformed an output cue (querying) and an input cue (accepting data),and at least arguably may be considered as an outputter and an inputter.

In addition, even though the apparatus 900 as shown includes a processor902, for certain embodiments a processor that is supporting theexecution, authoring, etc. of scripts may in fact be physically separatefrom the remainder of an apparatus; thus an apparatus may have noonboard processor, relying on a remote processor, cloud processing,etc., as facilitated for example by a communicator 908. It should beunderstood that other elements likewise may be distributed and/orabstracted; while the arrangement for the apparatus 900 shown ispresented for an explanation, a wide variety of different embodimentsmay serve the functions as described herein.

FIG. 10 shows a block diagram is shown of an apparatus 1000 that mayperform various operations, and store various information generatedand/or used by such operations, according to an embodiment. Theapparatus may represent any computer or processing system describedherein. The processing system 1000 is a hardware device on which any ofthe other entities, components, or services depicted in the examples ofFIG. 1 through FIG. 14 (and any other components described in thisspecification) may be implemented. The processing system 1000 includesone or more processors 1002 and memory 1004 coupled to an interconnect1012. The interconnect 1012 is shown in FIG. 15 as an abstraction thatrepresents any one or more separate physical buses, point to pointconnections, or both connected by appropriate bridges, adapters, orcontrollers. The interconnect 1012, therefore, may include, for example,a system bus, a Peripheral Component Interconnect (PCI) bus orPCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The processor(s) 1002 is/are the central processing unit of theprocessing system 1000 and, thus, control the overall operation of theprocessing system 1000. In certain embodiments, the processor(s) 1002accomplish this by executing software or firmware stored in memory 1004.The processor(s) 1002 may be, or may include, one or more programmablegeneral-purpose or special-purpose microprocessors, digital signalprocessors (DSPs), programmable controllers, application specificintegrated circuits (ASICs), programmable logic devices (PLDs), trustedplatform modules (TPMs), or the like, or a combination of such devices.

The memory 1004 is or includes the main memory of the processing system1000. The memory 1004 represents any form of random access memory (RAM),read-only memory (ROM), flash memory, or the like, or a combination ofsuch devices. In use, the memory 1004 may contain a code. In oneembodiment, the code includes a general programming module configured torecognize the general-purpose program received via the computer businterface and prepare the general-purpose program for execution at theprocessor. In another embodiment, the general programming module may beimplemented using hardware circuitry such as ASICs, PLDs, orfield-programmable gate arrays (FPGAs).

The network adapter 1010, a storage device(s) 1006, and I/O device(s)1008, are also connected to the processor(s) 1002 through theinterconnect 1012 The network adapter 1010 provides the processingsystem 1000 with the ability to communicate with remote devices over anetwork and may be, for example, an Ethernet adapter or Fibre Channeladapter. The network adapter 1010 may also provide the processing system1000 with the ability to communicate with other computers within thecluster. In some embodiments, the processing system 1000 may use morethan one network adapter to deal with the communications within andoutside of the cluster separately.

The I/O device(s) 1008 can include, for example, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The I/O device(s)1008 also may include, for example, cameras and/or other imagers adaptedto accept visual input including but not limited to postures and/orgestures. The display device may include, for example, a cathode raytube (CRT), liquid crystal display (LCD), or some other applicable knownor convenient display device. The display device may take various forms,including but not limited to stereo displays suited for use in near-eyeapplications such as head mounted displays or other wearable devices.

The code stored in memory 1004 may be implemented as software and/orfirmware to program the processor(s) 1002 to carry out actions describedherein. In certain embodiments, such software or firmware may beinitially provided to the processing system 1000 by downloading from aremote system through the processing system 1000 (e.g., via networkadapter 1010).

The techniques herein may be implemented by, for example, programmablecircuitry (e.g. one or more microprocessors) programmed with softwareand/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more ASICs, PLDs, FPGAs, etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine.

A machine can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch, or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

A machine-accessible storage medium or a storage device(s) 1006includes, for example, recordable/non-recordable media (e.g., ROM; RAM;magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc., or any combination thereof. The storage mediumtypically may be non-transitory or include a non-transitory device. Inthis context, a non-transitory storage medium may include a device thatis tangible, meaning that the device has a concrete physical form,although the device may change its physical state. Thus, for example,non-transitory refers to a device remaining tangible despite this changein state.

The term “logic”, as used herein, may include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theembodiment. Since many embodiments of the embodiment can be made withoutdeparting from the spirit and scope of the embodiments, the embodimentsresides in the claims hereinafter appended.

1. An apparatus, comprising: a housing adapted to be worn on a head ofan operator; a processing device coupled to the housing; a stereographical display in communication with the processing device andattached to the housing such that when the housing is worn by theoperator the stereo graphical display is facing and proximate eyes ofthe operator; and an input device in communication with the processingdevice and disposed on the housing, wherein the processing device is to:generate a script for a user-implemented procedure, the scriptcomprising: a graphical step avatar class and a step class associatedwith the graphical step avatar class; a graphical link avatar class anda link class associated with the graphical link avatar class; agraphical output cue avatar class and an output cue class associatedwith the graphical output cue avatar class; a graphical input cue avatarclass and an input cue class associated with the graphical input cueavatar class; instantiate a graphical step avatar from the graphicalstep avatar class instantiates; instantiate a graphical link avatar fromthe graphical link avatar class; instantiate a. graphical output cueavatar from the graphical output cue avatar class; and execute thescript corresponding to the user-implemented procedure.